2023-04-10 02:53:36,692 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089 2023-04-10 02:53:36,705 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-10 02:53:36,723 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-10 02:53:36,723 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/cluster_55f563f5-c06c-630a-24a5-87501b65e74d, deleteOnExit=true 2023-04-10 02:53:36,724 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-10 02:53:36,724 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/test.cache.data in system properties and HBase conf 2023-04-10 02:53:36,725 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/hadoop.tmp.dir in system properties and HBase conf 2023-04-10 02:53:36,725 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/hadoop.log.dir in system properties and HBase conf 2023-04-10 02:53:36,726 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-10 02:53:36,726 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-10 02:53:36,726 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-10 02:53:36,850 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-10 02:53:37,249 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-10 02:53:37,254 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-10 02:53:37,255 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-10 02:53:37,255 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-10 02:53:37,255 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-10 02:53:37,256 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-10 02:53:37,257 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-10 02:53:37,257 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-10 02:53:37,257 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-10 02:53:37,258 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-10 02:53:37,258 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/nfs.dump.dir in system properties and HBase conf 2023-04-10 02:53:37,259 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/java.io.tmpdir in system properties and HBase conf 2023-04-10 02:53:37,259 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-10 02:53:37,260 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-10 02:53:37,260 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-10 02:53:37,873 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-10 02:53:37,878 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-10 02:53:38,215 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-10 02:53:38,403 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-10 02:53:38,417 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-10 02:53:38,458 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-10 02:53:38,490 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/java.io.tmpdir/Jetty_localhost_37239_hdfs____a9c8gu/webapp 2023-04-10 02:53:38,635 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37239 2023-04-10 02:53:38,674 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-10 02:53:38,675 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-10 02:53:39,109 WARN [Listener at localhost/42593] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-10 02:53:39,241 WARN [Listener at localhost/42593] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-10 02:53:39,264 WARN [Listener at localhost/42593] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-10 02:53:39,270 INFO [Listener at localhost/42593] log.Slf4jLog(67): jetty-6.1.26 2023-04-10 02:53:39,277 INFO [Listener at localhost/42593] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/java.io.tmpdir/Jetty_localhost_44007_datanode____gfif4u/webapp 2023-04-10 02:53:39,402 INFO [Listener at localhost/42593] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44007 2023-04-10 02:53:39,788 WARN [Listener at localhost/44533] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-10 02:53:40,321 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x86901b8345edaf0a: Processing first storage report for DS-671fbeca-3db3-4a61-aa1e-9b19b9dc33a1 from datanode 3badec8f-43a8-47c0-ae8c-646406d4f0bc 2023-04-10 02:53:40,323 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x86901b8345edaf0a: from storage DS-671fbeca-3db3-4a61-aa1e-9b19b9dc33a1 node DatanodeRegistration(127.0.0.1:43623, datanodeUuid=3badec8f-43a8-47c0-ae8c-646406d4f0bc, infoPort=41767, infoSecurePort=0, ipcPort=44533, storageInfo=lv=-57;cid=testClusterID;nsid=901033609;c=1681095217954), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-10 02:53:40,323 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x86901b8345edaf0a: Processing first storage report for DS-6ffe5035-a77d-4d91-b6c0-0624b7eef6a3 from datanode 3badec8f-43a8-47c0-ae8c-646406d4f0bc 2023-04-10 02:53:40,323 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x86901b8345edaf0a: from storage DS-6ffe5035-a77d-4d91-b6c0-0624b7eef6a3 node DatanodeRegistration(127.0.0.1:43623, datanodeUuid=3badec8f-43a8-47c0-ae8c-646406d4f0bc, infoPort=41767, infoSecurePort=0, ipcPort=44533, storageInfo=lv=-57;cid=testClusterID;nsid=901033609;c=1681095217954), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-10 02:53:40,391 DEBUG [Listener at localhost/44533] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089 2023-04-10 02:53:40,458 INFO [Listener at localhost/44533] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/cluster_55f563f5-c06c-630a-24a5-87501b65e74d/zookeeper_0, clientPort=60568, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/cluster_55f563f5-c06c-630a-24a5-87501b65e74d/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/cluster_55f563f5-c06c-630a-24a5-87501b65e74d/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-10 02:53:40,472 INFO [Listener at localhost/44533] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60568 2023-04-10 02:53:40,482 INFO [Listener at localhost/44533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-10 02:53:40,484 INFO [Listener at localhost/44533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-10 02:53:41,159 INFO [Listener at localhost/44533] util.FSUtils(471): Created version file at hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717 with version=8 2023-04-10 02:53:41,160 INFO [Listener at localhost/44533] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/hbase-staging 2023-04-10 02:53:41,477 INFO [Listener at localhost/44533] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-10 02:53:41,949 INFO [Listener at localhost/44533] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-10 02:53:41,981 INFO [Listener at localhost/44533] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-10 02:53:41,982 INFO [Listener at localhost/44533] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-10 02:53:41,982 INFO [Listener at localhost/44533] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-10 02:53:41,982 INFO [Listener at localhost/44533] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-10 02:53:41,982 INFO [Listener at localhost/44533] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-10 02:53:42,125 INFO [Listener at localhost/44533] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-10 02:53:42,198 DEBUG [Listener at localhost/44533] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-10 02:53:42,294 INFO [Listener at localhost/44533] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36119 2023-04-10 02:53:42,305 INFO [Listener at localhost/44533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-10 02:53:42,307 INFO [Listener at localhost/44533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-10 02:53:42,328 INFO [Listener at localhost/44533] zookeeper.RecoverableZooKeeper(93): Process identifier=master:36119 connecting to ZooKeeper ensemble=127.0.0.1:60568 2023-04-10 02:53:42,371 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:361190x0, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-10 02:53:42,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:36119-0x1016bfd46a00000 connected 2023-04-10 02:53:42,401 DEBUG [Listener at localhost/44533] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-10 02:53:42,402 DEBUG [Listener at localhost/44533] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-10 02:53:42,405 DEBUG [Listener at localhost/44533] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-10 02:53:42,413 DEBUG [Listener at localhost/44533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36119 2023-04-10 02:53:42,413 DEBUG [Listener at localhost/44533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36119 2023-04-10 02:53:42,414 DEBUG [Listener at localhost/44533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36119 2023-04-10 02:53:42,415 DEBUG [Listener at localhost/44533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36119 2023-04-10 02:53:42,415 DEBUG [Listener at localhost/44533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36119 2023-04-10 02:53:42,422 INFO [Listener at localhost/44533] master.HMaster(444): hbase.rootdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717, hbase.cluster.distributed=false 2023-04-10 02:53:42,502 INFO [Listener at localhost/44533] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-10 02:53:42,502 INFO [Listener at localhost/44533] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-10 02:53:42,502 INFO [Listener at localhost/44533] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-10 02:53:42,503 INFO [Listener at localhost/44533] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-10 02:53:42,503 INFO [Listener at localhost/44533] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-10 02:53:42,503 INFO [Listener at localhost/44533] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-10 02:53:42,509 INFO [Listener at localhost/44533] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-10 02:53:42,512 INFO [Listener at localhost/44533] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38321 2023-04-10 02:53:42,514 INFO [Listener at localhost/44533] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-10 02:53:42,520 DEBUG [Listener at localhost/44533] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-10 02:53:42,521 INFO [Listener at localhost/44533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-10 02:53:42,524 INFO [Listener at localhost/44533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-10 02:53:42,526 INFO [Listener at localhost/44533] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38321 connecting to ZooKeeper ensemble=127.0.0.1:60568 2023-04-10 02:53:42,530 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:383210x0, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-10 02:53:42,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:38321-0x1016bfd46a00001 connected 2023-04-10 02:53:42,532 DEBUG [Listener at localhost/44533] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-10 02:53:42,533 DEBUG [Listener at localhost/44533] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-10 02:53:42,534 DEBUG [Listener at localhost/44533] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-10 02:53:42,535 DEBUG [Listener at localhost/44533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38321 2023-04-10 02:53:42,535 DEBUG [Listener at localhost/44533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38321 2023-04-10 02:53:42,536 DEBUG [Listener at localhost/44533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38321 2023-04-10 02:53:42,540 DEBUG [Listener at localhost/44533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38321 2023-04-10 02:53:42,540 DEBUG [Listener at localhost/44533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38321 2023-04-10 02:53:42,542 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,36119,1681095221303 2023-04-10 02:53:42,552 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-10 02:53:42,553 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,36119,1681095221303 2023-04-10 02:53:42,571 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-10 02:53:42,571 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-10 02:53:42,571 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-10 02:53:42,572 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-10 02:53:42,573 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,36119,1681095221303 from backup master directory 2023-04-10 02:53:42,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-10 02:53:42,576 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,36119,1681095221303 2023-04-10 02:53:42,576 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-10 02:53:42,577 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-10 02:53:42,577 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,36119,1681095221303 2023-04-10 02:53:42,580 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-10 02:53:42,581 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-10 02:53:42,677 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/hbase.id with ID: 1f0c1cea-bc92-4d79-8a56-434aa17e63b0 2023-04-10 02:53:42,719 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-10 02:53:42,735 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-10 02:53:42,776 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x6f4adb4d to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:53:42,810 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d8cb569, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:53:42,838 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-10 02:53:42,840 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-10 02:53:42,861 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-10 02:53:42,861 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-10 02:53:42,863 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:53:42,868 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:53:42,869 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-10 02:53:42,904 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/MasterData/data/master/store-tmp 2023-04-10 02:53:43,336 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:53:43,336 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-10 02:53:43,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-10 02:53:43,337 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-10 02:53:43,337 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-10 02:53:43,337 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-10 02:53:43,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-10 02:53:43,337 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-10 02:53:43,339 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/MasterData/WALs/jenkins-hbase4.apache.org,36119,1681095221303 2023-04-10 02:53:43,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36119%2C1681095221303, suffix=, logDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/MasterData/WALs/jenkins-hbase4.apache.org,36119,1681095221303, archiveDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/MasterData/oldWALs, maxLogs=10 2023-04-10 02:53:43,424 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43623,DS-671fbeca-3db3-4a61-aa1e-9b19b9dc33a1,DISK] 2023-04-10 02:53:43,433 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:53:43,503 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/MasterData/WALs/jenkins-hbase4.apache.org,36119,1681095221303/jenkins-hbase4.apache.org%2C36119%2C1681095221303.1681095223371 2023-04-10 02:53:43,505 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43623,DS-671fbeca-3db3-4a61-aa1e-9b19b9dc33a1,DISK]] 2023-04-10 02:53:43,506 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-10 02:53:43,507 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:53:43,511 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-10 02:53:43,512 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-10 02:53:43,582 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-10 02:53:43,591 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-10 02:53:43,621 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-10 02:53:43,633 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:53:43,639 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-10 02:53:43,641 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-10 02:53:43,659 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-10 02:53:43,664 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-10 02:53:43,665 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72704189, jitterRate=0.08337683975696564}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-10 02:53:43,665 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-10 02:53:43,667 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-10 02:53:43,688 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-10 02:53:43,688 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-10 02:53:43,690 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-10 02:53:43,692 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-10 02:53:43,724 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-04-10 02:53:43,724 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-10 02:53:43,750 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-10 02:53:43,756 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-10 02:53:43,780 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-10 02:53:43,784 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-10 02:53:43,786 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-10 02:53:43,793 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-10 02:53:43,798 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-10 02:53:43,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-10 02:53:43,803 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-10 02:53:43,804 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-10 02:53:43,819 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-10 02:53:43,823 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-10 02:53:43,824 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-10 02:53:43,824 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-10 02:53:43,824 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,36119,1681095221303, sessionid=0x1016bfd46a00000, setting cluster-up flag (Was=false) 2023-04-10 02:53:43,839 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-10 02:53:43,843 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-10 02:53:43,845 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,36119,1681095221303 2023-04-10 02:53:43,850 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-10 02:53:43,855 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-10 02:53:43,857 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,36119,1681095221303 2023-04-10 02:53:43,859 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.hbase-snapshot/.tmp 2023-04-10 02:53:43,945 INFO [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(952): ClusterId : 1f0c1cea-bc92-4d79-8a56-434aa17e63b0 2023-04-10 02:53:43,950 DEBUG [RS:0;jenkins-hbase4:38321] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-10 02:53:43,957 DEBUG [RS:0;jenkins-hbase4:38321] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-10 02:53:43,957 DEBUG [RS:0;jenkins-hbase4:38321] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-10 02:53:43,961 DEBUG [RS:0;jenkins-hbase4:38321] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-10 02:53:43,962 DEBUG [RS:0;jenkins-hbase4:38321] zookeeper.ReadOnlyZKClient(139): Connect 0x341e8eb4 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:53:43,967 DEBUG [RS:0;jenkins-hbase4:38321] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ad96647, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:53:43,968 DEBUG [RS:0;jenkins-hbase4:38321] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ee6e5c4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-10 02:53:43,977 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-10 02:53:43,990 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-10 02:53:43,990 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-10 02:53:43,990 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-10 02:53:43,990 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-10 02:53:43,991 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-10 02:53:43,991 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-10 02:53:43,991 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-10 02:53:43,991 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-10 02:53:43,993 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681095253993 2023-04-10 02:53:43,996 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-10 02:53:43,998 DEBUG [RS:0;jenkins-hbase4:38321] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:38321 2023-04-10 02:53:43,999 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-10 02:53:44,000 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-10 02:53:44,003 INFO [RS:0;jenkins-hbase4:38321] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-10 02:53:44,003 INFO [RS:0;jenkins-hbase4:38321] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-10 02:53:44,003 DEBUG [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1023): About to register with Master. 2023-04-10 02:53:44,006 INFO [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,36119,1681095221303 with isa=jenkins-hbase4.apache.org/172.31.14.131:38321, startcode=1681095222501 2023-04-10 02:53:44,006 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-10 02:53:44,008 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-10 02:53:44,018 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-10 02:53:44,019 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-10 02:53:44,019 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-10 02:53:44,019 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-10 02:53:44,020 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,021 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-10 02:53:44,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-10 02:53:44,024 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-10 02:53:44,028 DEBUG [RS:0;jenkins-hbase4:38321] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-10 02:53:44,028 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-10 02:53:44,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-10 02:53:44,031 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681095224031,5,FailOnTimeoutGroup] 2023-04-10 02:53:44,032 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681095224031,5,FailOnTimeoutGroup] 2023-04-10 02:53:44,032 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,032 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-10 02:53:44,034 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,035 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,050 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-10 02:53:44,052 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-10 02:53:44,052 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717 2023-04-10 02:53:44,074 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:53:44,078 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-10 02:53:44,082 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/info 2023-04-10 02:53:44,083 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-10 02:53:44,086 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:53:44,086 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-10 02:53:44,089 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/rep_barrier 2023-04-10 02:53:44,090 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-10 02:53:44,091 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:53:44,092 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-10 02:53:44,095 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/table 2023-04-10 02:53:44,095 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48375, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-10 02:53:44,096 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-10 02:53:44,097 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:53:44,099 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740 2023-04-10 02:53:44,101 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740 2023-04-10 02:53:44,105 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-10 02:53:44,108 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-10 02:53:44,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:44,114 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-10 02:53:44,115 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64684740, jitterRate=-0.036122262477874756}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-10 02:53:44,115 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-10 02:53:44,115 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-10 02:53:44,115 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-10 02:53:44,115 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-10 02:53:44,115 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-10 02:53:44,116 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-10 02:53:44,117 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-10 02:53:44,117 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-10 02:53:44,123 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-10 02:53:44,123 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-10 02:53:44,128 DEBUG [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717 2023-04-10 02:53:44,128 DEBUG [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42593 2023-04-10 02:53:44,128 DEBUG [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-10 02:53:44,132 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-10 02:53:44,135 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-10 02:53:44,135 DEBUG [RS:0;jenkins-hbase4:38321] zookeeper.ZKUtil(162): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:44,135 WARN [RS:0;jenkins-hbase4:38321] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-10 02:53:44,136 INFO [RS:0;jenkins-hbase4:38321] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-10 02:53:44,137 DEBUG [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/WALs/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:44,140 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,38321,1681095222501] 2023-04-10 02:53:44,150 DEBUG [RS:0;jenkins-hbase4:38321] zookeeper.ZKUtil(162): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:44,152 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-10 02:53:44,155 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-10 02:53:44,162 DEBUG [RS:0;jenkins-hbase4:38321] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-10 02:53:44,170 INFO [RS:0;jenkins-hbase4:38321] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-10 02:53:44,190 INFO [RS:0;jenkins-hbase4:38321] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-10 02:53:44,195 INFO [RS:0;jenkins-hbase4:38321] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-10 02:53:44,196 INFO [RS:0;jenkins-hbase4:38321] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,197 INFO [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-10 02:53:44,206 INFO [RS:0;jenkins-hbase4:38321] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,206 DEBUG [RS:0;jenkins-hbase4:38321] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-10 02:53:44,206 DEBUG [RS:0;jenkins-hbase4:38321] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-10 02:53:44,206 DEBUG [RS:0;jenkins-hbase4:38321] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-10 02:53:44,207 DEBUG [RS:0;jenkins-hbase4:38321] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-10 02:53:44,207 DEBUG [RS:0;jenkins-hbase4:38321] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-10 02:53:44,207 DEBUG [RS:0;jenkins-hbase4:38321] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-10 02:53:44,207 DEBUG [RS:0;jenkins-hbase4:38321] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-10 02:53:44,207 DEBUG [RS:0;jenkins-hbase4:38321] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-10 02:53:44,207 DEBUG [RS:0;jenkins-hbase4:38321] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-10 02:53:44,207 DEBUG [RS:0;jenkins-hbase4:38321] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-10 02:53:44,208 INFO [RS:0;jenkins-hbase4:38321] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,208 INFO [RS:0;jenkins-hbase4:38321] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,208 INFO [RS:0;jenkins-hbase4:38321] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,224 INFO [RS:0;jenkins-hbase4:38321] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-10 02:53:44,226 INFO [RS:0;jenkins-hbase4:38321] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38321,1681095222501-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,247 INFO [RS:0;jenkins-hbase4:38321] regionserver.Replication(203): jenkins-hbase4.apache.org,38321,1681095222501 started 2023-04-10 02:53:44,247 INFO [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,38321,1681095222501, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:38321, sessionid=0x1016bfd46a00001 2023-04-10 02:53:44,248 DEBUG [RS:0;jenkins-hbase4:38321] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-10 02:53:44,248 DEBUG [RS:0;jenkins-hbase4:38321] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:44,248 DEBUG [RS:0;jenkins-hbase4:38321] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38321,1681095222501' 2023-04-10 02:53:44,248 DEBUG [RS:0;jenkins-hbase4:38321] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:44,249 DEBUG [RS:0;jenkins-hbase4:38321] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:44,249 DEBUG [RS:0;jenkins-hbase4:38321] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-10 02:53:44,249 DEBUG [RS:0;jenkins-hbase4:38321] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-10 02:53:44,250 DEBUG [RS:0;jenkins-hbase4:38321] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:44,250 DEBUG [RS:0;jenkins-hbase4:38321] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38321,1681095222501' 2023-04-10 02:53:44,250 DEBUG [RS:0;jenkins-hbase4:38321] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-10 02:53:44,250 DEBUG [RS:0;jenkins-hbase4:38321] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-10 02:53:44,251 DEBUG [RS:0;jenkins-hbase4:38321] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-10 02:53:44,251 INFO [RS:0;jenkins-hbase4:38321] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-10 02:53:44,251 INFO [RS:0;jenkins-hbase4:38321] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-10 02:53:44,307 DEBUG [jenkins-hbase4:36119] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-10 02:53:44,309 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38321,1681095222501, state=OPENING 2023-04-10 02:53:44,316 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-10 02:53:44,318 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-10 02:53:44,318 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-10 02:53:44,322 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:53:44,363 INFO [RS:0;jenkins-hbase4:38321] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38321%2C1681095222501, suffix=, logDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/WALs/jenkins-hbase4.apache.org,38321,1681095222501, archiveDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/oldWALs, maxLogs=32 2023-04-10 02:53:44,383 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43623,DS-671fbeca-3db3-4a61-aa1e-9b19b9dc33a1,DISK] 2023-04-10 02:53:44,387 INFO [RS:0;jenkins-hbase4:38321] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/WALs/jenkins-hbase4.apache.org,38321,1681095222501/jenkins-hbase4.apache.org%2C38321%2C1681095222501.1681095224366 2023-04-10 02:53:44,387 DEBUG [RS:0;jenkins-hbase4:38321] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43623,DS-671fbeca-3db3-4a61-aa1e-9b19b9dc33a1,DISK]] 2023-04-10 02:53:44,506 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:44,509 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-10 02:53:44,512 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55564, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-10 02:53:44,523 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-10 02:53:44,524 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-10 02:53:44,527 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38321%2C1681095222501.meta, suffix=.meta, logDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/WALs/jenkins-hbase4.apache.org,38321,1681095222501, archiveDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/oldWALs, maxLogs=32 2023-04-10 02:53:44,543 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43623,DS-671fbeca-3db3-4a61-aa1e-9b19b9dc33a1,DISK] 2023-04-10 02:53:44,547 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/WALs/jenkins-hbase4.apache.org,38321,1681095222501/jenkins-hbase4.apache.org%2C38321%2C1681095222501.meta.1681095224528.meta 2023-04-10 02:53:44,548 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43623,DS-671fbeca-3db3-4a61-aa1e-9b19b9dc33a1,DISK]] 2023-04-10 02:53:44,548 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-10 02:53:44,550 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-10 02:53:44,565 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-10 02:53:44,570 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-10 02:53:44,576 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-10 02:53:44,576 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:53:44,576 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-10 02:53:44,576 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-10 02:53:44,579 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-10 02:53:44,581 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/info 2023-04-10 02:53:44,581 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/info 2023-04-10 02:53:44,582 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-10 02:53:44,582 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:53:44,583 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-10 02:53:44,584 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/rep_barrier 2023-04-10 02:53:44,584 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/rep_barrier 2023-04-10 02:53:44,585 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-10 02:53:44,586 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:53:44,586 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-10 02:53:44,587 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/table 2023-04-10 02:53:44,588 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/table 2023-04-10 02:53:44,588 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-10 02:53:44,589 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:53:44,591 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740 2023-04-10 02:53:44,593 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740 2023-04-10 02:53:44,597 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-10 02:53:44,599 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-10 02:53:44,600 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72255051, jitterRate=0.07668416202068329}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-10 02:53:44,600 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-10 02:53:44,610 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681095224498 2023-04-10 02:53:44,626 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-10 02:53:44,627 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-10 02:53:44,627 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38321,1681095222501, state=OPEN 2023-04-10 02:53:44,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-10 02:53:44,630 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-10 02:53:44,634 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-10 02:53:44,634 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38321,1681095222501 in 308 msec 2023-04-10 02:53:44,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-10 02:53:44,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 504 msec 2023-04-10 02:53:44,644 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 740 msec 2023-04-10 02:53:44,644 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681095224644, completionTime=-1 2023-04-10 02:53:44,644 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-10 02:53:44,645 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-10 02:53:44,705 DEBUG [hconnection-0x10f63bbb-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:53:44,709 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55574, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:53:44,731 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-10 02:53:44,731 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681095284731 2023-04-10 02:53:44,731 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681095344731 2023-04-10 02:53:44,731 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 86 msec 2023-04-10 02:53:44,755 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36119,1681095221303-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,755 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36119,1681095221303-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,755 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36119,1681095221303-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,757 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:36119, period=300000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,757 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-10 02:53:44,763 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-10 02:53:44,770 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-10 02:53:44,772 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-10 02:53:44,784 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-10 02:53:44,786 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-10 02:53:44,789 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-10 02:53:44,811 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/hbase/namespace/78f9d525c18aba75e4f5b4c3629f0ae0 2023-04-10 02:53:44,813 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/hbase/namespace/78f9d525c18aba75e4f5b4c3629f0ae0 empty. 2023-04-10 02:53:44,814 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/hbase/namespace/78f9d525c18aba75e4f5b4c3629f0ae0 2023-04-10 02:53:44,814 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-10 02:53:44,838 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-10 02:53:44,840 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 78f9d525c18aba75e4f5b4c3629f0ae0, NAME => 'hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp 2023-04-10 02:53:44,855 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:53:44,855 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 78f9d525c18aba75e4f5b4c3629f0ae0, disabling compactions & flushes 2023-04-10 02:53:44,855 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. 2023-04-10 02:53:44,855 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. 2023-04-10 02:53:44,855 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. after waiting 0 ms 2023-04-10 02:53:44,855 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. 2023-04-10 02:53:44,855 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. 2023-04-10 02:53:44,855 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 78f9d525c18aba75e4f5b4c3629f0ae0: 2023-04-10 02:53:44,860 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-10 02:53:44,875 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681095224863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095224863"}]},"ts":"1681095224863"} 2023-04-10 02:53:44,903 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-10 02:53:44,905 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-10 02:53:44,910 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095224905"}]},"ts":"1681095224905"} 2023-04-10 02:53:44,914 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-10 02:53:44,922 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=78f9d525c18aba75e4f5b4c3629f0ae0, ASSIGN}] 2023-04-10 02:53:44,925 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=78f9d525c18aba75e4f5b4c3629f0ae0, ASSIGN 2023-04-10 02:53:44,926 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=78f9d525c18aba75e4f5b4c3629f0ae0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38321,1681095222501; forceNewPlan=false, retain=false 2023-04-10 02:53:45,078 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=78f9d525c18aba75e4f5b4c3629f0ae0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:45,078 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681095225077"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095225077"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095225077"}]},"ts":"1681095225077"} 2023-04-10 02:53:45,083 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 78f9d525c18aba75e4f5b4c3629f0ae0, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:53:45,242 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. 2023-04-10 02:53:45,243 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 78f9d525c18aba75e4f5b4c3629f0ae0, NAME => 'hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0.', STARTKEY => '', ENDKEY => ''} 2023-04-10 02:53:45,244 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 78f9d525c18aba75e4f5b4c3629f0ae0 2023-04-10 02:53:45,244 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:53:45,244 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 78f9d525c18aba75e4f5b4c3629f0ae0 2023-04-10 02:53:45,244 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 78f9d525c18aba75e4f5b4c3629f0ae0 2023-04-10 02:53:45,248 INFO [StoreOpener-78f9d525c18aba75e4f5b4c3629f0ae0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 78f9d525c18aba75e4f5b4c3629f0ae0 2023-04-10 02:53:45,250 DEBUG [StoreOpener-78f9d525c18aba75e4f5b4c3629f0ae0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/namespace/78f9d525c18aba75e4f5b4c3629f0ae0/info 2023-04-10 02:53:45,250 DEBUG [StoreOpener-78f9d525c18aba75e4f5b4c3629f0ae0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/namespace/78f9d525c18aba75e4f5b4c3629f0ae0/info 2023-04-10 02:53:45,251 INFO [StoreOpener-78f9d525c18aba75e4f5b4c3629f0ae0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 78f9d525c18aba75e4f5b4c3629f0ae0 columnFamilyName info 2023-04-10 02:53:45,252 INFO [StoreOpener-78f9d525c18aba75e4f5b4c3629f0ae0-1] regionserver.HStore(310): Store=78f9d525c18aba75e4f5b4c3629f0ae0/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:53:45,253 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/namespace/78f9d525c18aba75e4f5b4c3629f0ae0 2023-04-10 02:53:45,254 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/namespace/78f9d525c18aba75e4f5b4c3629f0ae0 2023-04-10 02:53:45,258 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 78f9d525c18aba75e4f5b4c3629f0ae0 2023-04-10 02:53:45,262 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/namespace/78f9d525c18aba75e4f5b4c3629f0ae0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-10 02:53:45,263 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 78f9d525c18aba75e4f5b4c3629f0ae0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68288574, jitterRate=0.01757904887199402}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-10 02:53:45,263 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 78f9d525c18aba75e4f5b4c3629f0ae0: 2023-04-10 02:53:45,265 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0., pid=6, masterSystemTime=1681095225236 2023-04-10 02:53:45,268 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. 2023-04-10 02:53:45,269 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. 2023-04-10 02:53:45,270 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=78f9d525c18aba75e4f5b4c3629f0ae0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:45,270 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681095225269"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681095225269"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681095225269"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681095225269"}]},"ts":"1681095225269"} 2023-04-10 02:53:45,277 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-10 02:53:45,278 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 78f9d525c18aba75e4f5b4c3629f0ae0, server=jenkins-hbase4.apache.org,38321,1681095222501 in 192 msec 2023-04-10 02:53:45,281 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-10 02:53:45,281 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=78f9d525c18aba75e4f5b4c3629f0ae0, ASSIGN in 356 msec 2023-04-10 02:53:45,283 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-10 02:53:45,283 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095225283"}]},"ts":"1681095225283"} 2023-04-10 02:53:45,286 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-10 02:53:45,289 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-10 02:53:45,289 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-10 02:53:45,291 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-10 02:53:45,291 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-10 02:53:45,292 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 516 msec 2023-04-10 02:53:45,326 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-10 02:53:45,358 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-10 02:53:45,366 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 46 msec 2023-04-10 02:53:45,372 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-10 02:53:45,390 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-10 02:53:45,395 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 23 msec 2023-04-10 02:53:45,411 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-10 02:53:45,414 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-10 02:53:45,418 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.841sec 2023-04-10 02:53:45,420 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-10 02:53:45,422 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-10 02:53:45,422 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-10 02:53:45,424 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36119,1681095221303-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-10 02:53:45,425 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36119,1681095221303-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-10 02:53:45,436 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-10 02:53:45,451 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x3ddd91d3 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:53:45,457 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51f71277, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:53:45,470 DEBUG [hconnection-0x286751c0-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:53:45,482 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55576, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:53:45,493 INFO [Listener at localhost/44533] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,36119,1681095221303 2023-04-10 02:53:45,517 INFO [Listener at localhost/44533] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=208, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=407, ProcessCount=169, AvailableMemoryMB=2439 2023-04-10 02:53:45,534 DEBUG [Listener at localhost/44533] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-10 02:53:45,537 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39732, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-10 02:53:45,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-10 02:53:45,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-10 02:53:45,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-10 02:53:45,558 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-10 02:53:45,561 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-10 02:53:45,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-10 02:53:45,564 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:45,567 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595 empty. 2023-04-10 02:53:45,569 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:45,569 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-10 02:53:45,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-10 02:53:45,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-10 02:53:45,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-10 02:53:45,990 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-10 02:53:45,993 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 66e59774c088f69ab4d8d1711037d595, NAME => 'TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp 2023-04-10 02:53:46,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-10 02:53:46,406 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:53:46,407 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 66e59774c088f69ab4d8d1711037d595, disabling compactions & flushes 2023-04-10 02:53:46,407 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:46,407 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:46,407 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. after waiting 0 ms 2023-04-10 02:53:46,407 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:46,407 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:46,407 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:46,412 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-10 02:53:46,413 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095226413"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095226413"}]},"ts":"1681095226413"} 2023-04-10 02:53:46,416 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-10 02:53:46,417 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-10 02:53:46,418 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095226417"}]},"ts":"1681095226417"} 2023-04-10 02:53:46,420 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-10 02:53:46,424 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=66e59774c088f69ab4d8d1711037d595, ASSIGN}] 2023-04-10 02:53:46,427 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=66e59774c088f69ab4d8d1711037d595, ASSIGN 2023-04-10 02:53:46,428 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=66e59774c088f69ab4d8d1711037d595, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38321,1681095222501; forceNewPlan=false, retain=false 2023-04-10 02:53:46,580 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=66e59774c088f69ab4d8d1711037d595, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:46,580 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095226580"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095226580"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095226580"}]},"ts":"1681095226580"} 2023-04-10 02:53:46,583 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:53:46,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-10 02:53:46,743 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:46,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 66e59774c088f69ab4d8d1711037d595, NAME => 'TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.', STARTKEY => '', ENDKEY => ''} 2023-04-10 02:53:46,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:46,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:53:46,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:46,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:46,746 INFO [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:46,748 DEBUG [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A 2023-04-10 02:53:46,748 DEBUG [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A 2023-04-10 02:53:46,750 INFO [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:53:46,750 INFO [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 66e59774c088f69ab4d8d1711037d595 columnFamilyName A 2023-04-10 02:53:46,751 INFO [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] regionserver.HStore(310): Store=66e59774c088f69ab4d8d1711037d595/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:53:46,752 INFO [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:46,754 DEBUG [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B 2023-04-10 02:53:46,754 DEBUG [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B 2023-04-10 02:53:46,754 INFO [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:53:46,755 INFO [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 66e59774c088f69ab4d8d1711037d595 columnFamilyName B 2023-04-10 02:53:46,755 INFO [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] regionserver.HStore(310): Store=66e59774c088f69ab4d8d1711037d595/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:53:46,755 INFO [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:46,757 DEBUG [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C 2023-04-10 02:53:46,757 DEBUG [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C 2023-04-10 02:53:46,758 INFO [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:53:46,758 INFO [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 66e59774c088f69ab4d8d1711037d595 columnFamilyName C 2023-04-10 02:53:46,759 INFO [StoreOpener-66e59774c088f69ab4d8d1711037d595-1] regionserver.HStore(310): Store=66e59774c088f69ab4d8d1711037d595/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:53:46,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:46,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:46,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:46,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-10 02:53:46,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:46,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-10 02:53:46,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 66e59774c088f69ab4d8d1711037d595; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66532993, jitterRate=-0.008581146597862244}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-10 02:53:46,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:46,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., pid=11, masterSystemTime=1681095226737 2023-04-10 02:53:46,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:46,778 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:46,779 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=66e59774c088f69ab4d8d1711037d595, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:46,780 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095226779"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681095226779"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681095226779"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681095226779"}]},"ts":"1681095226779"} 2023-04-10 02:53:46,788 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-10 02:53:46,788 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 in 201 msec 2023-04-10 02:53:46,791 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-10 02:53:46,792 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=66e59774c088f69ab4d8d1711037d595, ASSIGN in 364 msec 2023-04-10 02:53:46,793 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-10 02:53:46,794 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095226793"}]},"ts":"1681095226793"} 2023-04-10 02:53:46,796 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-10 02:53:46,799 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-10 02:53:46,802 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2450 sec 2023-04-10 02:53:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-10 02:53:47,693 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-10 02:53:47,698 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x2678d495 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:53:47,703 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8c3f619, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:53:47,706 DEBUG [Listener at localhost/44533] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:53:47,709 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55588, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:53:47,712 DEBUG [Listener at localhost/44533] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-10 02:53:47,715 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39748, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-10 02:53:47,725 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x1b886c8f to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:53:47,730 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@748f869b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:53:47,730 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x0e16d41b to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:53:47,739 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e91d243, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:53:47,739 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x77648431 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:53:47,745 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30eb7fb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:53:47,745 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x7e728705 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:53:47,750 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46192d17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:53:47,750 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x09a575b8 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:53:47,759 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c57e1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:53:47,760 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x1818ad0a to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:53:47,766 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49d71120, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:53:47,767 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x04fb2b82 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:53:47,798 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59d6b6ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:53:47,799 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x6129209e to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:53:47,805 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d24bbb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:53:47,805 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x4b02eeb3 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:53:47,812 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63871a9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:53:47,824 DEBUG [hconnection-0x55d4bc5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:53:47,836 DEBUG [hconnection-0x7576e4ed-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:53:47,836 DEBUG [hconnection-0x62a93ad6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:53:47,839 DEBUG [hconnection-0x605ba4da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:53:47,843 DEBUG [hconnection-0x3cbe814d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:53:47,844 DEBUG [hconnection-0x1e045d2b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:53:47,844 DEBUG [hconnection-0x5ad720d2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:53:47,862 DEBUG [hconnection-0x7d721dea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:53:47,892 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55592, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:53:47,903 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55598, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:53:47,905 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:53:47,906 DEBUG [hconnection-0x156ee019-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:53:47,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:47,909 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55614, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:53:47,910 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55622, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:53:47,932 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55652, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:53:47,937 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55668, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:53:47,939 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55662, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:53:47,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:47,948 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:47,949 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:47,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:47,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:47,952 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55684, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:53:47,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:47,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:47,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:47,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:47,959 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:47,960 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:47,960 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:47,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:47,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:47,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:47,964 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:47,964 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:47,975 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:47,975 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:47,982 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:47,984 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:47,987 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-10 02:53:48,005 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:48,005 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:48,006 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:48,006 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:48,006 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:48,006 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:48,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:48,018 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:48,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095228081, max timestamp=1681095228119Num uniques 859; 2023-04-10 02:53:48,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:53:48,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681095228081, max timestamp=1681095228119Num uniques 859; 2023-04-10 02:53:48,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-10 02:53:48,190 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/8e8af2f0ccd54f9cb97bfe0a86fc0879 2023-04-10 02:53:48,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681095228081, max timestamp=1681095228119Num uniques 859; 2023-04-10 02:53:48,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-10 02:53:48,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681095228289, max timestamp=1681095228311Num uniques 1018; 2023-04-10 02:53:48,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681095228289, max timestamp=1681095228311Num uniques 1009; 2023-04-10 02:53:48,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-10 02:53:48,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-10 02:53:48,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681095228289, max timestamp=1681095228311Num uniques 1009; 2023-04-10 02:53:48,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-10 02:53:48,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095288361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095288365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681095228347, max timestamp=1681095228365Num uniques 1027; 2023-04-10 02:53:48,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-10 02:53:48,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681095228347, max timestamp=1681095228365Num uniques 1009; 2023-04-10 02:53:48,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46207, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-10 02:53:48,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681095228347, max timestamp=1681095228362Num uniques 1027; 2023-04-10 02:53:48,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-10 02:53:48,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095288425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681095228407, max timestamp=1681095228428Num uniques 1009; 2023-04-10 02:53:48,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-10 02:53:48,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681095228407, max timestamp=1681095228428Num uniques 1009; 2023-04-10 02:53:48,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46221, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-10 02:53:48,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8512, totalHeapSize=19584, min timestamp=1681095228407, max timestamp=1681095228429Num uniques 1009; 2023-04-10 02:53:48,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46210, new segment data size=8512, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-10 02:53:48,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681095228471, max timestamp=1681095228483Num uniques 1045; 2023-04-10 02:53:48,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-10 02:53:48,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681095228471, max timestamp=1681095228482Num uniques 1045; 2023-04-10 02:53:48,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-10 02:53:48,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681095228471, max timestamp=1681095228483Num uniques 1045; 2023-04-10 02:53:48,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-10 02:53:48,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095288529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095288530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681095228516, max timestamp=1681095228530Num uniques 1054; 2023-04-10 02:53:48,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-10 02:53:48,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681095228516, max timestamp=1681095228530Num uniques 1054; 2023-04-10 02:53:48,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-10 02:53:48,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681095228516, max timestamp=1681095228530Num uniques 1054; 2023-04-10 02:53:48,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-10 02:53:48,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095288571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095288573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095288574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095288642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095288642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095288679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095288682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095288682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,695 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/40a738e6d1cd42f0a3dcc80d75af610f 2023-04-10 02:53:48,794 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/decc60b4380b45b298d000a5788956fb 2023-04-10 02:53:48,805 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/8e8af2f0ccd54f9cb97bfe0a86fc0879 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8e8af2f0ccd54f9cb97bfe0a86fc0879 2023-04-10 02:53:48,819 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8e8af2f0ccd54f9cb97bfe0a86fc0879, entries=150, sequenceid=10, filesize=11.6 K 2023-04-10 02:53:48,832 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/40a738e6d1cd42f0a3dcc80d75af610f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/40a738e6d1cd42f0a3dcc80d75af610f 2023-04-10 02:53:48,853 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/40a738e6d1cd42f0a3dcc80d75af610f, entries=150, sequenceid=10, filesize=11.6 K 2023-04-10 02:53:48,856 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/decc60b4380b45b298d000a5788956fb as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/decc60b4380b45b298d000a5788956fb 2023-04-10 02:53:48,866 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/decc60b4380b45b298d000a5788956fb, entries=150, sequenceid=10, filesize=11.6 K 2023-04-10 02:53:48,870 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 66e59774c088f69ab4d8d1711037d595 in 883ms, sequenceid=10, compaction requested=false 2023-04-10 02:53:48,874 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-10 02:53:48,877 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:48,878 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:48,879 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:48,888 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:48,893 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:48,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:48,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b26aa60[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:48,898 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:48,898 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:48,898 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:48,898 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:48,902 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:48,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:48,902 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:48,902 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:48,902 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:48,916 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:48,916 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:48,916 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:48,920 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,921 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:48,921 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:48,921 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:48,921 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:48,922 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:48,922 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:48,922 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:48,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:48,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:48,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:48,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:48,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:48,934 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:48,937 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=147.60 KB heapSize=421.31 KB 2023-04-10 02:53:48,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:48,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:53:48,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:48,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:53:48,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:48,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:53:48,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,946 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,973 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,973 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:48,973 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:48,973 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,973 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:48,973 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:48,973 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:48,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:48,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:48,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:48,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:48,973 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:48,973 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:48,980 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:48,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:48,982 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:48,982 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:48,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095288947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095288950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095288954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:48,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:48,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095288955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095288997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,026 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/234552b740184d949b73be731d42f4f6 2023-04-10 02:53:49,083 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:49,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:49,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:49,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095289101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:49,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:49,128 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:49,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095289102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,128 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:49,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095289102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095289104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095289104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,154 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:49,154 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:49,155 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,155 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,160 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,161 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:49,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:49,165 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,165 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:49,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:49,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:49,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:49,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:49,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/f5e60bb30882429b80b864ace4d21991 2023-04-10 02:53:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:49,188 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:49,188 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:49,197 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:49,197 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:49,197 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:49,197 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:49,198 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:49,198 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:49,205 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,205 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:49,206 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,206 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:49,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:49,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d66652[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:49,210 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:49,210 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,218 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,218 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:49,218 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:49,218 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:49,218 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,218 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:49,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:49,225 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:49,225 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:49,225 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,247 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/939618c2269042788dd5b955022a9128 2023-04-10 02:53:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:49,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:49,248 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:49,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:49,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:49,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:49,249 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:49,253 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,253 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:49,254 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:49,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:49,255 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,258 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,261 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/234552b740184d949b73be731d42f4f6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/234552b740184d949b73be731d42f4f6 2023-04-10 02:53:49,271 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,272 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:49,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:49,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:49,272 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:49,272 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:49,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,273 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,273 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,273 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,273 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:49,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:49,273 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:49,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:49,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:49,282 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/234552b740184d949b73be731d42f4f6, entries=200, sequenceid=134, filesize=14.0 K 2023-04-10 02:53:49,283 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/f5e60bb30882429b80b864ace4d21991 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/f5e60bb30882429b80b864ace4d21991 2023-04-10 02:53:49,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/f5e60bb30882429b80b864ace4d21991, entries=150, sequenceid=134, filesize=11.7 K 2023-04-10 02:53:49,302 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/939618c2269042788dd5b955022a9128 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/939618c2269042788dd5b955022a9128 2023-04-10 02:53:49,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/939618c2269042788dd5b955022a9128, entries=150, sequenceid=134, filesize=11.7 K 2023-04-10 02:53:49,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~421.27 KB/431376, currentSize=33.54 KB/34350 for 66e59774c088f69ab4d8d1711037d595 in 382ms, sequenceid=134, compaction requested=false 2023-04-10 02:53:49,322 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:49,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:49,338 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:53:49,339 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:49,339 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:49,339 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:49,339 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:49,339 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:49,339 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:49,373 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:49,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:49,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:49,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:49,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:49,405 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:49,405 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:49,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095229377, max timestamp=1681095229394Num uniques 859; 2023-04-10 02:53:49,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095289382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:53:49,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095229377, max timestamp=1681095229394Num uniques 859; 2023-04-10 02:53:49,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095229377, max timestamp=1681095229393Num uniques 859; 2023-04-10 02:53:49,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:53:49,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095289380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,414 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:49,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:53:49,418 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:49,418 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,419 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,420 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:49,420 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:49,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:49,421 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,421 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:49,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:49,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:49,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:49,425 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:49,426 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:49,426 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:49,426 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:49,426 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:49,426 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:49,426 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:49,430 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:49,430 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:49,430 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,430 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:49,430 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:49,439 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,439 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:49,439 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,439 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:49,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:49,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15db9014[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:49,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:49,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,445 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,445 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:49,445 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:49,445 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:49,446 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,446 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:49,448 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:49,450 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:49,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:49,456 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:49,457 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:49,457 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:49,457 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:49,457 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:49,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,466 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:49,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,469 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,483 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:49,483 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:49,483 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,483 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:49,483 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:49,483 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,484 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,484 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,484 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:49,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:49,484 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:49,484 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:49,484 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:49,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:49,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:49,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:49,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095289490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095289498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095289500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095289516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095289520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,585 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:49,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:49,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:49,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095289606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:49,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:49,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095289611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,624 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:49,624 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:49,633 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:49,633 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:49,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,637 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:49,637 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:49,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,638 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095289635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,638 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:49,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:49,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:49,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:49,649 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:49,649 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:49,649 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:49,650 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:49,660 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:49,660 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:49,668 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:49,669 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,669 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:49,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:49,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7727936c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:49,679 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:49,679 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,681 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:49,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:49,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:49,683 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,683 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:49,702 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,702 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:49,702 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:49,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:49,713 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:49,713 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:49,713 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:49,713 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:49,713 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:49,717 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:49,717 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:49,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,717 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:49,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:49,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:49,719 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,721 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,728 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,728 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,728 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,728 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:49,728 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:49,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:49,728 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:49,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:49,728 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:49,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:49,728 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,729 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,729 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:49,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095289729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095289728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:49,730 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:49,730 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:49,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095289810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,830 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:49,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:49,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:49,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095289836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:49,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095289841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/c6410cd4221448ba995ca15cc97d6b82 2023-04-10 02:53:49,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:49,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:49,857 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:49,857 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:49,861 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:49,861 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:49,861 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,861 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,863 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,863 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:49,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:49,864 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,864 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:49,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:49,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:49,872 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:49,872 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:49,874 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:49,875 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:49,875 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:49,875 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:49,875 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:49,875 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:49,876 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,881 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:49,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:49,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:49,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@193c4242[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:49,886 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:49,886 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,889 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:49,889 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,889 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:49,891 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:49,892 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:49,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:49,901 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:49,901 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:49,901 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:49,901 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:49,901 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:49,903 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,903 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:49,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,903 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:49,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:49,905 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,913 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,920 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:49,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:49,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:49,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:49,921 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:49,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:49,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:49,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:49,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,922 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:49,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:49,922 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:49,923 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:49,922 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,023 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:50,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:50,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:50,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095290033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095290033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:50,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:50,041 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:50,041 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:50,045 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:50,045 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:50,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,047 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,047 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:50,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:50,048 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,048 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:50,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:50,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:50,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:50,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:50,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:50,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:50,053 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,053 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:50,053 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:50,053 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,054 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:50,054 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:50,054 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,054 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:50,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:50,057 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,057 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,058 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,058 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:50,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ced53c1[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:50,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:50,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,066 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,067 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:50,067 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:50,067 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:50,067 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,067 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:50,069 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:50,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:50,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:50,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:50,079 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:50,079 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:50,079 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:50,079 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:50,079 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:50,081 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,081 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,082 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:50,083 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,086 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,093 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:50,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:50,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:50,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,094 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:50,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:50,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:50,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:50,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:50,108 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:50,108 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:50,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095290113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095290140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095290144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,205 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-10 02:53:50,208 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:50,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:50,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:50,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:50,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:50,220 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:50,220 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:50,221 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:50,221 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:50,222 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,222 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,223 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:50,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:50,224 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,224 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,224 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:50,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:50,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:50,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:50,226 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:50,226 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:50,227 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:50,227 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:50,227 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:50,227 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,227 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:50,227 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:50,227 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:50,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:50,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:50,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:50,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,238 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:50,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:50,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@657f3d78[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:50,243 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:50,243 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,244 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,244 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:50,244 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:50,244 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:50,245 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,245 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:50,247 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:50,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:50,252 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:50,252 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:50,252 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:50,252 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:50,252 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:50,254 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,254 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:50,254 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,256 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,259 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:50,266 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:50,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:50,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:50,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:50,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:50,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,267 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,267 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,268 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:50,268 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:50,268 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:50,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:50,313 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-10 02:53:50,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/cb5979e2d96346f787740e5e249a7c3e 2023-04-10 02:53:50,316 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-10 02:53:50,366 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/2b831db704014380ae446ac99b78393a 2023-04-10 02:53:50,368 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:50,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:50,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:50,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:50,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:50,379 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:50,379 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:50,381 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:50,381 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:50,381 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,381 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,382 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/c6410cd4221448ba995ca15cc97d6b82 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/c6410cd4221448ba995ca15cc97d6b82 2023-04-10 02:53:50,384 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:50,384 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:50,385 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,385 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:50,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:50,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:50,387 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:50,387 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:50,388 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:50,395 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:50,395 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:50,395 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,395 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:50,395 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:50,396 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,396 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:50,396 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:50,396 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,396 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:50,396 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:50,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/c6410cd4221448ba995ca15cc97d6b82, entries=200, sequenceid=144, filesize=14.1 K 2023-04-10 02:53:50,398 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,398 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:50,399 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/cb5979e2d96346f787740e5e249a7c3e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/cb5979e2d96346f787740e5e249a7c3e 2023-04-10 02:53:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:50,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e886e48[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:50,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:50,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,404 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:50,405 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,405 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:50,406 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,407 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:50,407 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:50,407 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/cb5979e2d96346f787740e5e249a7c3e, entries=150, sequenceid=144, filesize=11.8 K 2023-04-10 02:53:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:50,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:50,416 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:50,416 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:50,416 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:50,416 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:50,416 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:50,419 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,419 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,419 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,420 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/2b831db704014380ae446ac99b78393a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/2b831db704014380ae446ac99b78393a 2023-04-10 02:53:50,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:50,420 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,423 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,434 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,434 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,434 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:50,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:50,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:50,436 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:50,434 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:50,437 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:50,437 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:50,436 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:50,437 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/2b831db704014380ae446ac99b78393a, entries=150, sequenceid=144, filesize=11.8 K 2023-04-10 02:53:50,437 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,438 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,438 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,438 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:50,440 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 66e59774c088f69ab4d8d1711037d595 in 1102ms, sequenceid=144, compaction requested=true 2023-04-10 02:53:50,440 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:50,448 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:50,448 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:53:50,448 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:53:50,448 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:53:50,448 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:53:50,454 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35908 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:53:50,457 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/B is initiating minor compaction (all files) 2023-04-10 02:53:50,457 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/B in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,457 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/40a738e6d1cd42f0a3dcc80d75af610f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/f5e60bb30882429b80b864ace4d21991, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/cb5979e2d96346f787740e5e249a7c3e] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=35.1 K 2023-04-10 02:53:50,459 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40688 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:53:50,459 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/A is initiating minor compaction (all files) 2023-04-10 02:53:50,459 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/A in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,460 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8e8af2f0ccd54f9cb97bfe0a86fc0879, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/234552b740184d949b73be731d42f4f6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/c6410cd4221448ba995ca15cc97d6b82] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=39.7 K 2023-04-10 02:53:50,460 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 40a738e6d1cd42f0a3dcc80d75af610f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681095227941 2023-04-10 02:53:50,462 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 8e8af2f0ccd54f9cb97bfe0a86fc0879, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681095227941 2023-04-10 02:53:50,465 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting f5e60bb30882429b80b864ace4d21991, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681095228565 2023-04-10 02:53:50,465 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 234552b740184d949b73be731d42f4f6, keycount=200, bloomtype=ROW, size=14.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681095228565 2023-04-10 02:53:50,465 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting cb5979e2d96346f787740e5e249a7c3e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681095228950 2023-04-10 02:53:50,473 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting c6410cd4221448ba995ca15cc97d6b82, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681095228950 2023-04-10 02:53:50,503 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#B#compaction#9 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:50,514 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#A#compaction#10 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:50,537 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:50,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:50,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:50,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:50,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:50,547 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:50,547 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:50,550 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:50,550 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:50,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681095229482, max timestamp=1681095230548Num uniques 1018; 2023-04-10 02:53:50,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-10 02:53:50,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:50,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681095229482, max timestamp=1681095230549Num uniques 1018; 2023-04-10 02:53:50,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-10 02:53:50,554 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,554 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7680, totalHeapSize=17712, min timestamp=1681095229482, max timestamp=1681095230549Num uniques 1018; 2023-04-10 02:53:50,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46610, new segment data size=7680, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-10 02:53:50,554 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=102.87 KB heapSize=292.20 KB 2023-04-10 02:53:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:50,557 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,557 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:50,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:50,558 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,558 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:50,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:50,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:50,567 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:50,567 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:50,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:50,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:50,576 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:50,576 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,576 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:50,577 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:50,577 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,577 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:50,577 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:50,577 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,577 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:50,577 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:50,580 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,580 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,580 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,580 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@555d0789[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:50,582 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:50,583 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,585 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,585 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,585 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:50,585 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:50,585 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:50,585 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:50,587 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:50,587 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:50,587 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:50,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,600 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/b868fe15411c4447b4ff1449750d1721 2023-04-10 02:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:50,602 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:50,602 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:50,602 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:50,602 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:50,602 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:50,605 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,605 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:50,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,609 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,612 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,615 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,619 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:50,619 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:50,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:50,619 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:50,620 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:50,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:50,620 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,626 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,626 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,627 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,627 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,627 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:50,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:50,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:50,627 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:50,627 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:50,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681095230609, max timestamp=1681095230640Num uniques 859; 2023-04-10 02:53:50,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681095230609, max timestamp=1681095230640Num uniques 859; 2023-04-10 02:53:50,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-10 02:53:50,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-10 02:53:50,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681095230609, max timestamp=1681095230640Num uniques 859; 2023-04-10 02:53:50,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-10 02:53:50,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095290664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095290666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095290667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095290668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095290668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,727 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:50,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:50,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:50,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:50,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:50,735 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:50,735 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:50,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:50,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:50,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:50,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,740 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,740 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:50,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:50,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:50,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:50,744 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:50,744 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:50,746 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:50,747 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,747 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:50,747 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:50,747 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:50,750 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@136825b8[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:50,755 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:50,755 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,756 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,756 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:50,756 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:50,756 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:50,757 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,757 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:50,759 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:50,759 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:50,759 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:50,759 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095290773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:50,775 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:50,775 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:50,775 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:50,775 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:50,775 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:50,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,780 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,780 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:50,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,782 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095290780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095290780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,785 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095290780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095290780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,793 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:50,793 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:50,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:50,794 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:50,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:50,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:50,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:50,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:50,795 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:50,795 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:50,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,795 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,795 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,795 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,795 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,895 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:50,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:50,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:50,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:50,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:50,902 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:50,902 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:50,903 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:50,903 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:50,904 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,904 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,905 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:50,905 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,906 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,906 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:50,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:50,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:50,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:50,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:50,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:50,914 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,914 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:50,914 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:50,914 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:50,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,916 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:50,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fe04e50[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:50,920 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:50,920 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,922 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:50,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:50,924 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:50,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:50,933 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:50,933 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:50,934 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:50,934 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:50,934 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:50,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:50,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,936 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:50,939 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:50,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:50,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:50,951 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/e900bbf601124deb9745d61e8be82617 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/e900bbf601124deb9745d61e8be82617 2023-04-10 02:53:50,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:50,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:50,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:50,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:50,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:50,954 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:50,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:50,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:50,960 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:50,960 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:50,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:50,979 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/B of 66e59774c088f69ab4d8d1711037d595 into e900bbf601124deb9745d61e8be82617(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:50,980 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:50,980 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/B, priority=13, startTime=1681095230448; duration=0sec 2023-04-10 02:53:50,980 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:53:50,981 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:53:50,982 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/29abaa7cf546448ca92cb1f2660f7393 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/29abaa7cf546448ca92cb1f2660f7393 2023-04-10 02:53:50,983 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35908 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:53:50,983 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/C is initiating minor compaction (all files) 2023-04-10 02:53:50,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:50,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095290977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:50,983 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/C in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:50,983 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/decc60b4380b45b298d000a5788956fb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/939618c2269042788dd5b955022a9128, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/2b831db704014380ae446ac99b78393a] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=35.1 K 2023-04-10 02:53:50,985 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting decc60b4380b45b298d000a5788956fb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681095227941 2023-04-10 02:53:50,987 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 939618c2269042788dd5b955022a9128, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681095228565 2023-04-10 02:53:50,988 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 2b831db704014380ae446ac99b78393a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681095228950 2023-04-10 02:53:51,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095290988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095290987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095290989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095291002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,035 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/A of 66e59774c088f69ab4d8d1711037d595 into 29abaa7cf546448ca92cb1f2660f7393(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:51,035 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:51,035 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/A, priority=13, startTime=1681095230443; duration=0sec 2023-04-10 02:53:51,036 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:51,042 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/d7b3338fba0d4de7bb88a710d0015f9c 2023-04-10 02:53:51,046 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#C#compaction#13 average throughput is 0.44 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:51,066 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:51,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:51,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:51,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:51,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:51,096 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:51,096 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:51,097 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:51,097 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:51,097 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,097 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:51,101 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:51,101 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:51,102 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:51,102 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:51,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:51,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:51,111 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/aab38db9ee724421ac61dd162876fa3b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/aab38db9ee724421ac61dd162876fa3b 2023-04-10 02:53:51,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:51,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:51,113 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:51,113 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:51,114 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:51,114 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:51,114 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:51,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/56faf2c1bdd842d9971e2766faee2b93 2023-04-10 02:53:51,114 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:51,114 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:51,115 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:51,115 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:51,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:51,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:51,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:51,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:51,119 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,119 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:51,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:51,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62f79bca[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:51,123 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:51,123 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,125 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,125 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:51,126 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:51,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:51,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/b868fe15411c4447b4ff1449750d1721 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/b868fe15411c4447b4ff1449750d1721 2023-04-10 02:53:51,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:51,128 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/C of 66e59774c088f69ab4d8d1711037d595 into aab38db9ee724421ac61dd162876fa3b(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:51,128 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:51,128 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/C, priority=13, startTime=1681095230448; duration=0sec 2023-04-10 02:53:51,128 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:51,129 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:51,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:51,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:51,137 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:51,137 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:51,137 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:51,137 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:51,137 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:51,140 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,140 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:51,140 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:51,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:51,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:51,142 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,145 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:51,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:51,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:51,150 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:51,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:51,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:51,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:51,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:51,160 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:51,160 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,155 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/b868fe15411c4447b4ff1449750d1721, entries=250, sequenceid=183, filesize=16.4 K 2023-04-10 02:53:51,160 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,161 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,161 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,160 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:51,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:51,163 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/d7b3338fba0d4de7bb88a710d0015f9c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/d7b3338fba0d4de7bb88a710d0015f9c 2023-04-10 02:53:51,175 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/d7b3338fba0d4de7bb88a710d0015f9c, entries=150, sequenceid=183, filesize=11.8 K 2023-04-10 02:53:51,185 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/56faf2c1bdd842d9971e2766faee2b93 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/56faf2c1bdd842d9971e2766faee2b93 2023-04-10 02:53:51,197 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/56faf2c1bdd842d9971e2766faee2b93, entries=150, sequenceid=183, filesize=11.8 K 2023-04-10 02:53:51,200 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=161.02 KB/164880 for 66e59774c088f69ab4d8d1711037d595 in 646ms, sequenceid=183, compaction requested=false 2023-04-10 02:53:51,200 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:51,261 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:51,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:51,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:51,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:51,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:51,268 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:51,268 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:51,269 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:51,269 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:51,270 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,270 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:51,271 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:51,271 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:51,272 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,272 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:51,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:51,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:51,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:51,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:51,276 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:51,276 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:51,280 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:51,281 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:51,281 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:51,281 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:51,281 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:51,282 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=161.02 KB heapSize=441.23 KB 2023-04-10 02:53:51,282 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:51,282 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:51,282 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:51,282 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:51,282 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:51,283 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:51,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:51,291 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:51,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095291316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095291316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095291366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095291366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095291366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095291468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,473 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-10 02:53:51,473 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-10 02:53:51,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095291474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095291475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095291475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095291671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095291695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095291696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:51,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095291699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,716 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/ef21551b225e43b0acbac4e9fb521bc8 2023-04-10 02:53:51,758 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/24738c31b4464a5b80765965acb9ac75 2023-04-10 02:53:51,792 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/4bb9c16123ae40dba44ac4e3652232b4 2023-04-10 02:53:51,812 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/ef21551b225e43b0acbac4e9fb521bc8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/ef21551b225e43b0acbac4e9fb521bc8 2023-04-10 02:53:51,822 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/ef21551b225e43b0acbac4e9fb521bc8, entries=150, sequenceid=222, filesize=11.8 K 2023-04-10 02:53:51,826 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/24738c31b4464a5b80765965acb9ac75 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/24738c31b4464a5b80765965acb9ac75 2023-04-10 02:53:51,837 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/24738c31b4464a5b80765965acb9ac75, entries=150, sequenceid=222, filesize=11.8 K 2023-04-10 02:53:51,841 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/4bb9c16123ae40dba44ac4e3652232b4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/4bb9c16123ae40dba44ac4e3652232b4 2023-04-10 02:53:51,861 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/4bb9c16123ae40dba44ac4e3652232b4, entries=150, sequenceid=222, filesize=11.8 K 2023-04-10 02:53:51,864 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~441.19 KB/451776, currentSize=40.25 KB/41220 for 66e59774c088f69ab4d8d1711037d595 in 581ms, sequenceid=222, compaction requested=true 2023-04-10 02:53:51,864 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:51,865 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:51,865 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:51,865 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:51,868 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,868 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:51,870 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,870 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:51,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:51,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:51,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f2ba952[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:51,879 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:51,880 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:51,881 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:53:51,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:51,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:51,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:51,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:51,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:51,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:51,884 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,884 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,884 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:51,884 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:51,884 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:51,885 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:51,887 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:51,897 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:51,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:51,902 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:51,902 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:51,902 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:51,902 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:51,902 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:51,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:51,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:51,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:51,905 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:51,910 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,918 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:51,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:51,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:51,918 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,919 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:51,919 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:51,919 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:51,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:51,919 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:51,919 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:51,919 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,919 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:51,919 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:51,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:51,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:51,919 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:51,919 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:51,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:51,949 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/573808cadbd449b29a960ca24604a7bf 2023-04-10 02:53:51,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095231973, max timestamp=1681095231990Num uniques 859; 2023-04-10 02:53:51,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:53:51,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095231973, max timestamp=1681095231990Num uniques 859; 2023-04-10 02:53:51,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095231973, max timestamp=1681095231990Num uniques 859; 2023-04-10 02:53:51,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:53:51,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:53:52,009 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/8acb20a965ff461f97b03af3b8e8ed8a 2023-04-10 02:53:52,020 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:52,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:52,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:52,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095292025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095292027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095292032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095292033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:52,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095292033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:52,040 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:52,040 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:52,045 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:52,045 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:52,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:52,051 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:52,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:52,052 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,052 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,053 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:52,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:52,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:52,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:52,076 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:52,086 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:52,086 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:52,086 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:52,087 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:52,090 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,090 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:52,090 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:52,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:52,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d3c02ea[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:52,095 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:52,095 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:52,107 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,107 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:52,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,110 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:52,111 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:52,111 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:52,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:52,121 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/a43bfcca62a6474bad18fdc0a49b39bd 2023-04-10 02:53:52,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:52,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:52,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:52,123 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:52,123 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:52,123 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:52,123 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:52,123 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:52,128 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,128 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:52,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,128 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:52,131 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:52,133 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/573808cadbd449b29a960ca24604a7bf as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/573808cadbd449b29a960ca24604a7bf 2023-04-10 02:53:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,135 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095292136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:52,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:52,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:52,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:52,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:52,141 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:52,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:52,142 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:52,142 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,148 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,148 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,148 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,144 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:52,148 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:52,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095292137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:52,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095292141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,152 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/573808cadbd449b29a960ca24604a7bf, entries=150, sequenceid=232, filesize=11.8 K 2023-04-10 02:53:52,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/8acb20a965ff461f97b03af3b8e8ed8a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/8acb20a965ff461f97b03af3b8e8ed8a 2023-04-10 02:53:52,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095292151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095292152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/8acb20a965ff461f97b03af3b8e8ed8a, entries=150, sequenceid=232, filesize=11.8 K 2023-04-10 02:53:52,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/a43bfcca62a6474bad18fdc0a49b39bd as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/a43bfcca62a6474bad18fdc0a49b39bd 2023-04-10 02:53:52,203 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/a43bfcca62a6474bad18fdc0a49b39bd, entries=150, sequenceid=232, filesize=11.8 K 2023-04-10 02:53:52,208 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 66e59774c088f69ab4d8d1711037d595 in 327ms, sequenceid=232, compaction requested=true 2023-04-10 02:53:52,209 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:52,209 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:52,209 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:53:52,209 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:53:52,209 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:53:52,209 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:53:52,216 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48247 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:53:52,216 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/B is initiating minor compaction (all files) 2023-04-10 02:53:52,217 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/B in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:52,217 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/e900bbf601124deb9745d61e8be82617, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/d7b3338fba0d4de7bb88a710d0015f9c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/24738c31b4464a5b80765965acb9ac75, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/8acb20a965ff461f97b03af3b8e8ed8a] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=47.1 K 2023-04-10 02:53:52,217 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53027 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:53:52,217 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/A is initiating minor compaction (all files) 2023-04-10 02:53:52,217 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/A in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:52,217 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/29abaa7cf546448ca92cb1f2660f7393, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/b868fe15411c4447b4ff1449750d1721, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/ef21551b225e43b0acbac4e9fb521bc8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/573808cadbd449b29a960ca24604a7bf] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=51.8 K 2023-04-10 02:53:52,218 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting e900bbf601124deb9745d61e8be82617, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681095228950 2023-04-10 02:53:52,220 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 29abaa7cf546448ca92cb1f2660f7393, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681095228950 2023-04-10 02:53:52,229 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting b868fe15411c4447b4ff1449750d1721, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1681095229482 2023-04-10 02:53:52,230 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d7b3338fba0d4de7bb88a710d0015f9c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1681095229482 2023-04-10 02:53:52,230 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ef21551b225e43b0acbac4e9fb521bc8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681095230662 2023-04-10 02:53:52,231 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 573808cadbd449b29a960ca24604a7bf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681095231318 2023-04-10 02:53:52,231 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 24738c31b4464a5b80765965acb9ac75, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681095230662 2023-04-10 02:53:52,233 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 8acb20a965ff461f97b03af3b8e8ed8a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681095231318 2023-04-10 02:53:52,254 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:52,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:52,257 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#B#compaction#22 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:52,258 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#A#compaction#21 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:52,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:52,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:52,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:52,262 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:52,262 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:52,263 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:52,263 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:52,264 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,264 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:52,265 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:52,265 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:52,266 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,266 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:52,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:52,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:52,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:52,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:52,270 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:52,270 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:52,273 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:52,278 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:52,279 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:52,279 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:52,279 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:52,280 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-10 02:53:52,280 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:52,280 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:52,280 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:52,280 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:52,280 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:52,280 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:52,301 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/ccdb3aeb7aca43b8995fa829ec56c799 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/ccdb3aeb7aca43b8995fa829ec56c799 2023-04-10 02:53:52,323 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/A of 66e59774c088f69ab4d8d1711037d595 into ccdb3aeb7aca43b8995fa829ec56c799(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:52,323 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:52,323 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/A, priority=12, startTime=1681095232209; duration=0sec 2023-04-10 02:53:52,323 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:53:52,323 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:53:52,325 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/eb5b2fd9463e460fa87c71c863283b0e 2023-04-10 02:53:52,326 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/91592184a3ab417fbd74f90a39c6e64f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/91592184a3ab417fbd74f90a39c6e64f 2023-04-10 02:53:52,329 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48247 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:53:52,329 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/C is initiating minor compaction (all files) 2023-04-10 02:53:52,329 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/C in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:52,330 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/aab38db9ee724421ac61dd162876fa3b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/56faf2c1bdd842d9971e2766faee2b93, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/4bb9c16123ae40dba44ac4e3652232b4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/a43bfcca62a6474bad18fdc0a49b39bd] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=47.1 K 2023-04-10 02:53:52,331 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting aab38db9ee724421ac61dd162876fa3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681095228950 2023-04-10 02:53:52,337 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 56faf2c1bdd842d9971e2766faee2b93, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1681095229482 2023-04-10 02:53:52,338 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 4bb9c16123ae40dba44ac4e3652232b4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681095230662 2023-04-10 02:53:52,339 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting a43bfcca62a6474bad18fdc0a49b39bd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681095231318 2023-04-10 02:53:52,341 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/B of 66e59774c088f69ab4d8d1711037d595 into 91592184a3ab417fbd74f90a39c6e64f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:52,345 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:52,345 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/B, priority=12, startTime=1681095232209; duration=0sec 2023-04-10 02:53:52,345 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:52,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:52,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:52,369 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#C#compaction#25 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:52,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095292367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095292368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095292369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095292373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095292373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,416 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/43b3200facea4d33a5e52e34ac82c396 2023-04-10 02:53:52,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095292482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095292486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095292486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095292487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095292684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095292687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095292690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095292690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:52,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095292693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,859 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/b5b6e31ad7b947568d5028faac55cc20 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/b5b6e31ad7b947568d5028faac55cc20 2023-04-10 02:53:52,868 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/e5a74b886555424e8320d04ff29d2528 2023-04-10 02:53:52,869 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/C of 66e59774c088f69ab4d8d1711037d595 into b5b6e31ad7b947568d5028faac55cc20(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:52,869 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:52,869 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/C, priority=12, startTime=1681095232209; duration=0sec 2023-04-10 02:53:52,869 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:52,879 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/eb5b2fd9463e460fa87c71c863283b0e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/eb5b2fd9463e460fa87c71c863283b0e 2023-04-10 02:53:52,887 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/eb5b2fd9463e460fa87c71c863283b0e, entries=150, sequenceid=269, filesize=11.9 K 2023-04-10 02:53:52,889 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/43b3200facea4d33a5e52e34ac82c396 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/43b3200facea4d33a5e52e34ac82c396 2023-04-10 02:53:52,899 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/43b3200facea4d33a5e52e34ac82c396, entries=150, sequenceid=269, filesize=11.9 K 2023-04-10 02:53:52,900 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/e5a74b886555424e8320d04ff29d2528 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e5a74b886555424e8320d04ff29d2528 2023-04-10 02:53:52,911 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e5a74b886555424e8320d04ff29d2528, entries=150, sequenceid=269, filesize=11.9 K 2023-04-10 02:53:52,918 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for 66e59774c088f69ab4d8d1711037d595 in 638ms, sequenceid=269, compaction requested=false 2023-04-10 02:53:52,918 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:52,919 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:52,919 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:52,922 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:52,923 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,923 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:52,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:52,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a35fd3[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:52,926 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:52,926 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,930 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,930 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:52,930 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:52,930 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:52,931 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,931 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:52,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:52,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:52,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:52,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:52,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:52,937 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:52,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:52,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:52,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:52,937 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:52,940 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,940 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:52,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,940 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:52,942 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:52,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:52,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:52,943 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,946 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,946 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:52,947 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:52,947 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:52,947 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:52,947 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,947 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:52,947 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:52,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:52,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:52,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:52,947 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:52,947 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:52,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:52,948 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:52,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:52,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:52,948 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:52,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:52,993 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:53:52,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:52,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:52,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:52,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:52,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:52,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:53,017 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/51b56cf3ba4841a391099b6e27bc7661 2023-04-10 02:53:53,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095233016, max timestamp=1681095233022Num uniques 859; 2023-04-10 02:53:53,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:53:53,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095233016, max timestamp=1681095233022Num uniques 859; 2023-04-10 02:53:53,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:53:53,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095233016, max timestamp=1681095233022Num uniques 859; 2023-04-10 02:53:53,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:53:53,051 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:53,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:53,055 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/62e9359c85df42e49ab0a740328acc3f 2023-04-10 02:53:53,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:53,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:53,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:53,059 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:53,059 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:53,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:53,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:53,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:53,063 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:53,063 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:53,063 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:53,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:53,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:53,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:53,065 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:53,065 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:53,067 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:53,067 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:53,067 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:53,067 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:53,067 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:53,067 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:53,067 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:53,067 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:53,067 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:53,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,069 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:53,070 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,070 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ed4bdc6[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:53,072 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:53,072 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,074 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:53,074 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:53,074 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:53,076 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:53,076 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:53,076 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:53,081 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,085 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:53,085 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:53,085 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:53,085 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:53,085 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:53,085 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:53,085 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:53,086 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/4976a657d95645718d249af43000ac08 2023-04-10 02:53:53,088 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,089 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:53,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,089 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:53,091 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,096 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/51b56cf3ba4841a391099b6e27bc7661 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/51b56cf3ba4841a391099b6e27bc7661 2023-04-10 02:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,104 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,104 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:53,104 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:53,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:53,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:53,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:53,104 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:53,104 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:53,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:53,104 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:53,104 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,108 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,108 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:53,108 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:53,108 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:53,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:53,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095293100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,111 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/51b56cf3ba4841a391099b6e27bc7661, entries=150, sequenceid=283, filesize=11.9 K 2023-04-10 02:53:53,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095293102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,113 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/62e9359c85df42e49ab0a740328acc3f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/62e9359c85df42e49ab0a740328acc3f 2023-04-10 02:53:53,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095293103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095293109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/62e9359c85df42e49ab0a740328acc3f, entries=150, sequenceid=283, filesize=11.9 K 2023-04-10 02:53:53,129 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/4976a657d95645718d249af43000ac08 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/4976a657d95645718d249af43000ac08 2023-04-10 02:53:53,147 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/4976a657d95645718d249af43000ac08, entries=150, sequenceid=283, filesize=11.9 K 2023-04-10 02:53:53,152 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 66e59774c088f69ab4d8d1711037d595 in 159ms, sequenceid=283, compaction requested=true 2023-04-10 02:53:53,152 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:53,152 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:53:53,152 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:53,152 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:53:53,153 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:53,153 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:53:53,154 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:53:53,154 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/A is initiating minor compaction (all files) 2023-04-10 02:53:53,154 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/A in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:53,155 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/ccdb3aeb7aca43b8995fa829ec56c799, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/eb5b2fd9463e460fa87c71c863283b0e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/51b56cf3ba4841a391099b6e27bc7661] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=35.8 K 2023-04-10 02:53:53,155 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:53:53,155 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/B is initiating minor compaction (all files) 2023-04-10 02:53:53,155 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/B in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:53,155 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/91592184a3ab417fbd74f90a39c6e64f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/43b3200facea4d33a5e52e34ac82c396, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/62e9359c85df42e49ab0a740328acc3f] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=35.8 K 2023-04-10 02:53:53,156 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ccdb3aeb7aca43b8995fa829ec56c799, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681095231318 2023-04-10 02:53:53,156 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 91592184a3ab417fbd74f90a39c6e64f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681095231318 2023-04-10 02:53:53,159 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting eb5b2fd9463e460fa87c71c863283b0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1681095232023 2023-04-10 02:53:53,160 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 43b3200facea4d33a5e52e34ac82c396, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1681095232023 2023-04-10 02:53:53,161 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 51b56cf3ba4841a391099b6e27bc7661, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1681095232362 2023-04-10 02:53:53,161 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 62e9359c85df42e49ab0a740328acc3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1681095232362 2023-04-10 02:53:53,179 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#B#compaction#30 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:53,179 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#A#compaction#31 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:53,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:53,195 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-10 02:53:53,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:53,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:53,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:53,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:53,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:53,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:53,213 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:53,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:53,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:53,222 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/de5454143f5049fd8b7d54caa211257c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/de5454143f5049fd8b7d54caa211257c 2023-04-10 02:53:53,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:53,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:53,225 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:53,225 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:53,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:53,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:53,229 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/869c206eaba946c1a6754ea065e2862a 2023-04-10 02:53:53,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:53,231 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:53,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:53,231 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/8b1c360ca9d14090bfef37d53ce8a4d6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8b1c360ca9d14090bfef37d53ce8a4d6 2023-04-10 02:53:53,232 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,232 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:53,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:53,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095293224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095293230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:53,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:53,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095293230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095293231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095293233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:53,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:53,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:53,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:53,251 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:53,254 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:53,254 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:53,254 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:53,254 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:53,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:53,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:53,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:53,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:53,257 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:53,257 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/B of 66e59774c088f69ab4d8d1711037d595 into de5454143f5049fd8b7d54caa211257c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:53,257 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:53,257 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/B, priority=13, startTime=1681095233152; duration=0sec 2023-04-10 02:53:53,257 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:53:53,258 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:53:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:53,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:53,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cb7ee77[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:53,260 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,260 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:53,262 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:53:53,262 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/C is initiating minor compaction (all files) 2023-04-10 02:53:53,263 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/C in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:53,263 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/b5b6e31ad7b947568d5028faac55cc20, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e5a74b886555424e8320d04ff29d2528, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/4976a657d95645718d249af43000ac08] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=35.8 K 2023-04-10 02:53:53,264 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting b5b6e31ad7b947568d5028faac55cc20, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681095231318 2023-04-10 02:53:53,264 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,264 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,270 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,277 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting e5a74b886555424e8320d04ff29d2528, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1681095232023 2023-04-10 02:53:53,277 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:53,277 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:53,277 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:53,278 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 4976a657d95645718d249af43000ac08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1681095232362 2023-04-10 02:53:53,280 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:53,280 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:53,280 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:53,284 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:53,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:53,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:53,284 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/A of 66e59774c088f69ab4d8d1711037d595 into 8b1c360ca9d14090bfef37d53ce8a4d6(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:53,285 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:53,285 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/A, priority=13, startTime=1681095233152; duration=0sec 2023-04-10 02:53:53,285 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:53,293 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:53,293 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:53,293 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:53,293 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:53,293 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:53,297 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,297 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:53,297 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:53,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:53,299 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:53,302 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,308 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:53,308 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:53,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:53,308 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#C#compaction#34 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:53,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:53,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:53,309 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,310 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:53,310 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:53,310 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:53,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:53,310 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,310 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:53,310 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,310 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:53,310 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:53,310 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:53,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/fadbb6c8127c4bb29eefacdde2940763 2023-04-10 02:53:53,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095293337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095293337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095293338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095293338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,342 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/e3da6c5c20b84b11aa57c75a531d0ac0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e3da6c5c20b84b11aa57c75a531d0ac0 2023-04-10 02:53:53,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095293339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,356 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/C of 66e59774c088f69ab4d8d1711037d595 into e3da6c5c20b84b11aa57c75a531d0ac0(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:53,356 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:53,356 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/C, priority=13, startTime=1681095233153; duration=0sec 2023-04-10 02:53:53,356 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:53,366 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/d69ab199d099478c834df910c42e0698 2023-04-10 02:53:53,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/869c206eaba946c1a6754ea065e2862a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/869c206eaba946c1a6754ea065e2862a 2023-04-10 02:53:53,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/869c206eaba946c1a6754ea065e2862a, entries=200, sequenceid=320, filesize=14.3 K 2023-04-10 02:53:53,381 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/fadbb6c8127c4bb29eefacdde2940763 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/fadbb6c8127c4bb29eefacdde2940763 2023-04-10 02:53:53,389 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/fadbb6c8127c4bb29eefacdde2940763, entries=150, sequenceid=320, filesize=11.9 K 2023-04-10 02:53:53,391 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/d69ab199d099478c834df910c42e0698 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/d69ab199d099478c834df910c42e0698 2023-04-10 02:53:53,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/d69ab199d099478c834df910c42e0698, entries=150, sequenceid=320, filesize=11.9 K 2023-04-10 02:53:53,410 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for 66e59774c088f69ab4d8d1711037d595 in 214ms, sequenceid=320, compaction requested=false 2023-04-10 02:53:53,410 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:53,410 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:53,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:53,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:53,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:53,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:53,417 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:53,417 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:53,418 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:53,418 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:53,419 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,419 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:53,421 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:53,421 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:53,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:53,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:53,422 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,422 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:53,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:53,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:53,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:53,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:53,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:53,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:53,428 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:53,428 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:53,433 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:53,433 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:53,433 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:53,433 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:53,434 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:53,434 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:53:53,435 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:53,435 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:53,435 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:53,435 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:53,435 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:53,435 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:53,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:53,543 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:53,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095233586, max timestamp=1681095233589Num uniques 859; 2023-04-10 02:53:53,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:53:53,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095233586, max timestamp=1681095233589Num uniques 859; 2023-04-10 02:53:53,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:53:53,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095233586, max timestamp=1681095233589Num uniques 859; 2023-04-10 02:53:53,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:53:53,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095293612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095293615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095293616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095293617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095293620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095293722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095293723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55662 deadline: 1681095293723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095293723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095293725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,860 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/dd8edfc2f00d4582b8c537e2826d94b8 2023-04-10 02:53:53,877 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/31e316c6a9e24496a3dc824e462eb313 2023-04-10 02:53:53,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095293926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095293930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095293931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095293931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:53,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:53,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095293931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095294230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095294235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095294237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095294237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095294238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,293 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/e6765b6aa96b49fa96af34297234fcb5 2023-04-10 02:53:54,301 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/dd8edfc2f00d4582b8c537e2826d94b8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/dd8edfc2f00d4582b8c537e2826d94b8 2023-04-10 02:53:54,309 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/dd8edfc2f00d4582b8c537e2826d94b8, entries=150, sequenceid=334, filesize=11.9 K 2023-04-10 02:53:54,310 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/31e316c6a9e24496a3dc824e462eb313 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/31e316c6a9e24496a3dc824e462eb313 2023-04-10 02:53:54,337 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/31e316c6a9e24496a3dc824e462eb313, entries=150, sequenceid=334, filesize=11.9 K 2023-04-10 02:53:54,338 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/e6765b6aa96b49fa96af34297234fcb5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e6765b6aa96b49fa96af34297234fcb5 2023-04-10 02:53:54,348 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e6765b6aa96b49fa96af34297234fcb5, entries=150, sequenceid=334, filesize=11.9 K 2023-04-10 02:53:54,349 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 66e59774c088f69ab4d8d1711037d595 in 915ms, sequenceid=334, compaction requested=true 2023-04-10 02:53:54,349 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:54,349 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:54,349 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:54,352 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:54,353 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:54,354 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:54,354 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:54,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:54,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31719c59[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:54,360 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:54,360 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:54,361 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:54,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:54,362 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,362 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:54,363 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:54,365 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:54,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:54,368 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:54,368 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:54,368 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:54,368 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:54,368 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:54,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:54,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:54,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:54,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:54,376 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:54,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:54,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:54,379 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:54,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:54,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:54,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:54,382 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:54,381 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:54,382 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:54,382 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:54,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:54,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:54,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:54,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:54,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:54,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:54,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:54,483 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:54,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:54,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:54,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:54,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:54,493 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:54,493 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:54,494 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:54,494 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:54,495 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:54,495 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:54,496 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:54,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:54,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:54,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:54,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:54,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:54,499 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:54,499 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:54,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:54,500 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:54,500 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:54,500 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-10 02:53:54,500 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:54,500 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:54,500 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:54,500 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:54,500 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:54,501 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:53:54,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:54,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:54,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55614 deadline: 1681095294748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095294753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095294753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095294754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095294754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095294855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095294859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095294860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:54,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095294860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095294860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:54,912 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/8be63199820144f4ab4839bfdf1b7b4d 2023-04-10 02:53:55,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095295062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095295063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095295064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095295064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095295065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,332 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/3acc6353436249b29f19910f4f0b3a4f 2023-04-10 02:53:55,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095295369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095295370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55622 deadline: 1681095295370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095295370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095295370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,754 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/91a10dd4daf04638aa6dff5b4c4cafc5 2023-04-10 02:53:55,762 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/8be63199820144f4ab4839bfdf1b7b4d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8be63199820144f4ab4839bfdf1b7b4d 2023-04-10 02:53:55,771 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8be63199820144f4ab4839bfdf1b7b4d, entries=150, sequenceid=370, filesize=11.9 K 2023-04-10 02:53:55,772 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/3acc6353436249b29f19910f4f0b3a4f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/3acc6353436249b29f19910f4f0b3a4f 2023-04-10 02:53:55,779 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/3acc6353436249b29f19910f4f0b3a4f, entries=150, sequenceid=370, filesize=11.9 K 2023-04-10 02:53:55,781 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/91a10dd4daf04638aa6dff5b4c4cafc5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/91a10dd4daf04638aa6dff5b4c4cafc5 2023-04-10 02:53:55,790 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/91a10dd4daf04638aa6dff5b4c4cafc5, entries=150, sequenceid=370, filesize=11.9 K 2023-04-10 02:53:55,792 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 66e59774c088f69ab4d8d1711037d595 in 1292ms, sequenceid=370, compaction requested=true 2023-04-10 02:53:55,792 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:55,792 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:55,792 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:55,793 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:55,793 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,793 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:55,793 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,796 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:55,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:55,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b361063[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:55,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:55,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,800 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:55,802 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,802 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:55,805 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:55,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:55,810 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:55,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:55,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:55,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:55,810 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:55,813 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,813 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,813 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:55,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:55,821 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:55,824 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:55,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:55,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:55,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:55,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:55,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,829 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:55,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:55,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:55,832 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:55,832 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:55,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:55,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:55,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:53:55,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:55,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:55,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:55,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:55,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:55,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:55,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095295900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095295900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095295902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55592 deadline: 1681095295905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:55,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095295906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,912 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/2abc1bfb5b3b4032a858b2473c549e62 2023-04-10 02:53:55,932 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:55,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:55,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:55,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:55,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:55,948 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:55,948 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:55,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:55,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:55,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,951 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:55,952 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:55,952 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:55,952 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,952 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:55,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:55,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:55,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:55,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:55,954 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:55,954 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:55,956 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:55,957 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:55,957 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:55,957 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:55,957 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:55,957 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/528dadaef65643638bc2949006c8afc3 2023-04-10 02:53:55,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,959 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:55,959 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,960 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:55,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:55,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:55,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f7b24e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:55,963 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:55,963 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,965 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:55,966 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:55,966 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:55,966 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:55,968 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,968 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:55,970 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:55,970 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:55,970 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:55,970 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:55,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:55,986 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:55,986 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:55,986 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:55,986 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:55,986 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:55,988 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,988 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:55,988 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:55,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,989 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:55,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:55,993 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:56,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:56,003 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:56,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:56,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:56,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,004 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:56,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:56,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:56,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:56,004 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:56,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:56,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:56,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:56,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:56,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:56,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:56,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:56,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095296017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,027 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/91f5854ef9984ba0b1af7fa6a846f85f 2023-04-10 02:53:56,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095296018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095296018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095296021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095296022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/2abc1bfb5b3b4032a858b2473c549e62 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2abc1bfb5b3b4032a858b2473c549e62 2023-04-10 02:53:56,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2abc1bfb5b3b4032a858b2473c549e62, entries=200, sequenceid=382, filesize=14.3 K 2023-04-10 02:53:56,052 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/528dadaef65643638bc2949006c8afc3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/528dadaef65643638bc2949006c8afc3 2023-04-10 02:53:56,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/528dadaef65643638bc2949006c8afc3, entries=150, sequenceid=382, filesize=11.9 K 2023-04-10 02:53:56,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/91f5854ef9984ba0b1af7fa6a846f85f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/91f5854ef9984ba0b1af7fa6a846f85f 2023-04-10 02:53:56,077 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/91f5854ef9984ba0b1af7fa6a846f85f, entries=150, sequenceid=382, filesize=11.9 K 2023-04-10 02:53:56,078 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 66e59774c088f69ab4d8d1711037d595 in 200ms, sequenceid=382, compaction requested=true 2023-04-10 02:53:56,078 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:56,079 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:56,079 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:53:56,079 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:53:56,079 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:56,079 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:53:56,082 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61272 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:53:56,082 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/B is initiating minor compaction (all files) 2023-04-10 02:53:56,082 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/B in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:56,083 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/de5454143f5049fd8b7d54caa211257c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/fadbb6c8127c4bb29eefacdde2940763, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/31e316c6a9e24496a3dc824e462eb313, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/3acc6353436249b29f19910f4f0b3a4f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/528dadaef65643638bc2949006c8afc3] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=59.8 K 2023-04-10 02:53:56,083 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66152 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:53:56,083 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/A is initiating minor compaction (all files) 2023-04-10 02:53:56,083 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/A in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:56,083 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8b1c360ca9d14090bfef37d53ce8a4d6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/869c206eaba946c1a6754ea065e2862a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/dd8edfc2f00d4582b8c537e2826d94b8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8be63199820144f4ab4839bfdf1b7b4d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2abc1bfb5b3b4032a858b2473c549e62] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=64.6 K 2023-04-10 02:53:56,085 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 8b1c360ca9d14090bfef37d53ce8a4d6, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1681095232362 2023-04-10 02:53:56,086 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting de5454143f5049fd8b7d54caa211257c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1681095232362 2023-04-10 02:53:56,087 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 869c206eaba946c1a6754ea065e2862a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681095233074 2023-04-10 02:53:56,088 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting fadbb6c8127c4bb29eefacdde2940763, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681095233101 2023-04-10 02:53:56,088 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting dd8edfc2f00d4582b8c537e2826d94b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1681095233222 2023-04-10 02:53:56,089 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 8be63199820144f4ab4839bfdf1b7b4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1681095233610 2023-04-10 02:53:56,090 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 31e316c6a9e24496a3dc824e462eb313, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1681095233222 2023-04-10 02:53:56,090 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 2abc1bfb5b3b4032a858b2473c549e62, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1681095234749 2023-04-10 02:53:56,090 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 3acc6353436249b29f19910f4f0b3a4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1681095233610 2023-04-10 02:53:56,091 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 528dadaef65643638bc2949006c8afc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1681095234752 2023-04-10 02:53:56,104 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:56,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:56,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:56,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:56,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:56,117 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:56,117 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:56,117 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:56,117 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:56,117 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:56,117 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:56,119 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:56,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:56,119 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:56,119 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#A#compaction#45 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:56,120 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,120 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:56,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:56,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:56,130 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#B#compaction#46 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:56,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:56,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:56,140 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:56,140 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:56,140 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:56,152 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:56,152 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:56,152 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:56,152 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:56,152 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-10 02:53:56,154 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:56,154 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:56,154 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:56,155 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:56,155 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:56,155 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:56,183 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/f6340540963648bf85a81d2ffa867cd2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/f6340540963648bf85a81d2ffa867cd2 2023-04-10 02:53:56,190 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/a0c5aaafde274aac88811b17006e30d2 2023-04-10 02:53:56,190 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/A of 66e59774c088f69ab4d8d1711037d595 into f6340540963648bf85a81d2ffa867cd2(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:56,191 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:56,191 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/A, priority=11, startTime=1681095236078; duration=0sec 2023-04-10 02:53:56,191 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:53:56,191 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:53:56,194 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61272 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:53:56,194 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/C is initiating minor compaction (all files) 2023-04-10 02:53:56,194 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/C in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:56,194 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e3da6c5c20b84b11aa57c75a531d0ac0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/d69ab199d099478c834df910c42e0698, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e6765b6aa96b49fa96af34297234fcb5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/91a10dd4daf04638aa6dff5b4c4cafc5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/91f5854ef9984ba0b1af7fa6a846f85f] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=59.8 K 2023-04-10 02:53:56,194 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting e3da6c5c20b84b11aa57c75a531d0ac0, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1681095232362 2023-04-10 02:53:56,195 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting d69ab199d099478c834df910c42e0698, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681095233101 2023-04-10 02:53:56,196 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting e6765b6aa96b49fa96af34297234fcb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1681095233222 2023-04-10 02:53:56,197 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 91a10dd4daf04638aa6dff5b4c4cafc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1681095233610 2023-04-10 02:53:56,197 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 91f5854ef9984ba0b1af7fa6a846f85f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1681095234752 2023-04-10 02:53:56,209 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/6016299517cd4523940a8172fbaedcfb 2023-04-10 02:53:56,215 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#C#compaction#49 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:56,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:56,230 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:56,242 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/0bb99e10d5c240998c690fea08d9c473 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/0bb99e10d5c240998c690fea08d9c473 2023-04-10 02:53:56,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095296241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095296243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095296243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,250 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/C of 66e59774c088f69ab4d8d1711037d595 into 0bb99e10d5c240998c690fea08d9c473(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:56,250 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:56,250 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/C, priority=11, startTime=1681095236079; duration=0sec 2023-04-10 02:53:56,250 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:56,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095296248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095296249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095296350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095296350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095296351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095296353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095296354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095296555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55636 deadline: 1681095296556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095296560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095296560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095296562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,578 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/fe031d88692b47deb17111dd2a1c66b6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/fe031d88692b47deb17111dd2a1c66b6 2023-04-10 02:53:56,586 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/B of 66e59774c088f69ab4d8d1711037d595 into fe031d88692b47deb17111dd2a1c66b6(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:56,586 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:56,586 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/B, priority=11, startTime=1681095236079; duration=0sec 2023-04-10 02:53:56,587 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:56,634 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/7b427f992bf54ff0acc2448e6461cbff 2023-04-10 02:53:56,643 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/a0c5aaafde274aac88811b17006e30d2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a0c5aaafde274aac88811b17006e30d2 2023-04-10 02:53:56,650 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a0c5aaafde274aac88811b17006e30d2, entries=150, sequenceid=402, filesize=11.9 K 2023-04-10 02:53:56,652 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/6016299517cd4523940a8172fbaedcfb as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6016299517cd4523940a8172fbaedcfb 2023-04-10 02:53:56,659 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6016299517cd4523940a8172fbaedcfb, entries=150, sequenceid=402, filesize=11.9 K 2023-04-10 02:53:56,660 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/7b427f992bf54ff0acc2448e6461cbff as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/7b427f992bf54ff0acc2448e6461cbff 2023-04-10 02:53:56,668 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/7b427f992bf54ff0acc2448e6461cbff, entries=150, sequenceid=402, filesize=11.9 K 2023-04-10 02:53:56,669 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 66e59774c088f69ab4d8d1711037d595 in 517ms, sequenceid=402, compaction requested=false 2023-04-10 02:53:56,669 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:56,669 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:56,669 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:56,671 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:56,671 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,671 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:56,671 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:56,675 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:56,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:56,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d02a71[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:56,678 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:56,679 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:56,680 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:56,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:56,682 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,682 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:56,682 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:56,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:56,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:56,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:56,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:56,686 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:56,686 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:56,686 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:56,686 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:56,686 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:56,689 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:56,689 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:56,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:56,689 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:56,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:56,691 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,693 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:56,700 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:56,700 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:56,700 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:56,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:56,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:56,700 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:56,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:56,700 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:56,701 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:56,701 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,701 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:56,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:56,701 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:56,701 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:56,701 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:56,701 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:56,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:56,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:56,801 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:56,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:56,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:56,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:56,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:56,811 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:56,811 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:56,811 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:56,811 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:56,811 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:56,811 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:56,813 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:56,813 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:56,813 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:56,814 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:56,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:56,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:56,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:56,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:56,820 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:56,820 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:56,820 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:56,821 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:56,821 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:56,822 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:56,822 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:56,823 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-10 02:53:56,823 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:56,823 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:56,823 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:56,823 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:56,825 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:56,825 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:56,864 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:56,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:56,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095296881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095296882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095296885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095296887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095296888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095296990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095296991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095296992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095296992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:56,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:56,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095296995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:57,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095297194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:57,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:57,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095297196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095297198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:57,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095297200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:57,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095297200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,291 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/66e90230189e4c6396622355fb7165ff 2023-04-10 02:53:57,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:57,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095297497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:57,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095297500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:57,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:57,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095297501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095297503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:57,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095297503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,705 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/ab763019e6444286a6666bbc5be47bc2 2023-04-10 02:53:57,757 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/547e4ee6b5164432878370720cf03ee5 2023-04-10 02:53:57,764 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/66e90230189e4c6396622355fb7165ff as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/66e90230189e4c6396622355fb7165ff 2023-04-10 02:53:57,773 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/66e90230189e4c6396622355fb7165ff, entries=150, sequenceid=418, filesize=11.9 K 2023-04-10 02:53:57,774 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/ab763019e6444286a6666bbc5be47bc2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/ab763019e6444286a6666bbc5be47bc2 2023-04-10 02:53:57,781 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/ab763019e6444286a6666bbc5be47bc2, entries=150, sequenceid=418, filesize=11.9 K 2023-04-10 02:53:57,782 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/547e4ee6b5164432878370720cf03ee5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/547e4ee6b5164432878370720cf03ee5 2023-04-10 02:53:57,787 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/547e4ee6b5164432878370720cf03ee5, entries=150, sequenceid=418, filesize=11.9 K 2023-04-10 02:53:57,788 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 66e59774c088f69ab4d8d1711037d595 in 966ms, sequenceid=418, compaction requested=true 2023-04-10 02:53:57,789 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:57,789 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:57,789 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:57,792 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:57,792 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,793 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:57,793 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:57,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:57,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:57,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:57,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:57,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:57,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:57,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:57,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e7efb06[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:57,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:57,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:57,798 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:57,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:57,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:57,802 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:57,805 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:57,805 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:57,805 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:57,805 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:57,805 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:57,807 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:57,807 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:57,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:57,807 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:57,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:57,808 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:57,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,815 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:57,815 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:57,816 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:57,816 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:57,816 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:57,816 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,816 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:57,816 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:57,816 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:57,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:57,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:57,816 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:57,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:57,816 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:57,816 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:57,917 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:57,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:57,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:57,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:57,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:57,921 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:57,921 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:57,922 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:57,922 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:57,922 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:57,922 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:57,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,923 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:57,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:57,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:57,924 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:57,924 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:57,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:57,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:57,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:57,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:57,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:57,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:57,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:57,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:57,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:57,925 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:57,926 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:57,926 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-10 02:53:57,926 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:57,926 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:57,926 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:57,926 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:57,926 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:57,926 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:58,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:58,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:58,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095298014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095298015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095298018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095298018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095298019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095298120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095298121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095298121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095298121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095298124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095298323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095298323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095298324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095298324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095298326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,359 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/7215e69ca87244fb92668c9c774de9a5 2023-04-10 02:53:58,422 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/9d4d2bd708e742f79431023b97887e6d 2023-04-10 02:53:58,480 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/1acfca52020142189f2316a13c85b598 2023-04-10 02:53:58,488 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/7215e69ca87244fb92668c9c774de9a5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/7215e69ca87244fb92668c9c774de9a5 2023-04-10 02:53:58,496 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/7215e69ca87244fb92668c9c774de9a5, entries=150, sequenceid=437, filesize=11.9 K 2023-04-10 02:53:58,508 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/9d4d2bd708e742f79431023b97887e6d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/9d4d2bd708e742f79431023b97887e6d 2023-04-10 02:53:58,524 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/9d4d2bd708e742f79431023b97887e6d, entries=150, sequenceid=437, filesize=11.9 K 2023-04-10 02:53:58,525 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/1acfca52020142189f2316a13c85b598 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/1acfca52020142189f2316a13c85b598 2023-04-10 02:53:58,537 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/1acfca52020142189f2316a13c85b598, entries=150, sequenceid=437, filesize=11.9 K 2023-04-10 02:53:58,539 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 66e59774c088f69ab4d8d1711037d595 in 613ms, sequenceid=437, compaction requested=true 2023-04-10 02:53:58,539 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:58,539 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:58,539 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:58,541 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:58,541 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,541 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:58,541 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:58,543 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:58,543 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,543 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:58,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15d057a2[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:58,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:58,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,547 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:58,547 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:58,547 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:58,547 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,547 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:58,549 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:58,549 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:58,549 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:58,549 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:58,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:58,552 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:58,553 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:58,553 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:58,553 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:58,553 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:58,557 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,557 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:58,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,557 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:58,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,559 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,561 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,564 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:58,564 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:58,564 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:58,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:58,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:58,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:58,564 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,564 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,564 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:58,564 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,564 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:58,564 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,565 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,565 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:58,565 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:58,630 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-10 02:53:58,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:58,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:58,631 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:58,631 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:58,631 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:58,631 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:58,665 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:58,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:58,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:58,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:58,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:58,670 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:58,670 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:58,670 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:58,670 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:58,671 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,671 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,672 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,672 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:58,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:58,673 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,673 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:58,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:58,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:58,677 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:58,677 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:58,677 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:58,677 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:58,677 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:58,677 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:58,677 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,677 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:58,677 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:58,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:58,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095298676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:58,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095298678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,682 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:58,682 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7858e294[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:58,684 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:58,685 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,685 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:58,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095298682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,686 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095298681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:58,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:58,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095298682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:58,689 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:58,689 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:58,689 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:58,689 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:58,689 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:58,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:58,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:58,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,692 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,693 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,699 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,699 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,699 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:58,699 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:58,699 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:58,699 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,699 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:58,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:58,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:58,699 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:58,699 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,699 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,699 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:58,700 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:58,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:58,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095298783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095298783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095298787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095298788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095298788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,800 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:58,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:58,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:58,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:58,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:58,804 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:58,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:58,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:58,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:58,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,805 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:58,805 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,806 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,806 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:58,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:58,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:58,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:58,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:58,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:58,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:58,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:58,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:58,808 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:58,808 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:58,808 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:58,808 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:58,808 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:58,808 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:58,808 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,808 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:58,808 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:58,810 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:58,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:58,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:58,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae9cc4c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:58,813 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:58,813 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,814 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:58,814 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:58,814 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:58,814 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,814 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:58,816 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:58,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:58,819 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:58,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:58,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:58,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:58,819 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:58,821 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,821 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,822 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,822 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:58,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,833 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:58,835 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:58,835 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:58,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:58,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:58,836 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,836 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,836 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,836 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,836 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:53:58,836 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:58,836 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:58,836 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,836 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:58,836 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:58,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:58,937 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:58,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:58,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:58,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:58,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:58,941 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:58,941 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:58,941 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:58,941 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:58,941 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,941 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,943 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:58,943 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:58,943 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,943 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:58,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:58,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:58,945 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:58,945 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:58,945 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:58,945 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:58,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:58,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:58,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:58,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:58,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3336a8dc[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:58,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:58,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,951 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:58,951 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,951 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:58,953 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:58,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:58,957 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:58,957 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:58,957 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:58,957 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:58,957 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:58,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:58,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:58,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:58,959 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,961 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:58,965 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:58,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:58,965 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:58,965 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:58,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:58,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:58,965 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,965 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,965 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:58,965 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:58,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:58,965 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:58,965 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:58,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095298986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095298986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095298992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095298992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:58,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:58,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095298993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,047 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/2ec25b07a2f34a14b0e8943ff59cbde2 2023-04-10 02:53:59,065 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:59,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:59,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/3301f68c06af4ad8895f4fe177e429f7 2023-04-10 02:53:59,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:59,073 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:59,073 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:59,073 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:59,073 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:59,075 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:59,075 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:59,076 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,076 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:59,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:59,077 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,077 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:59,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:59,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:59,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:59,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:59,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:59,082 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:59,082 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:59,082 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:59,092 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:59,092 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:59,092 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:59,092 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:59,092 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:59,092 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:59,092 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:53:59,092 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:53:59,092 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,092 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:53:59,092 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:53:59,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,094 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:59,095 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:59,095 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:53:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:59,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:53:59,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@244c845f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:53:59,097 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:53:59,097 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:59,100 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:59,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:59,100 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,100 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:53:59,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:53:59,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:53:59,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:53:59,104 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,104 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:53:59,104 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:59,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:53:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:59,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:59,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:53:59,109 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:53:59,109 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:53:59,110 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:53:59,110 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:53:59,110 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:53:59,111 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:59,112 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:59,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:59,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:59,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:59,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:59,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:59,113 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/0add03d448ce49a4a7f835b71a03dbb1 2023-04-10 02:53:59,113 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:53:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:53:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:53:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:59,115 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:53:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:53:59,119 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:53:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/2ec25b07a2f34a14b0e8943ff59cbde2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2ec25b07a2f34a14b0e8943ff59cbde2 2023-04-10 02:53:59,123 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,123 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:59,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:59,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:59,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:53:59,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:59,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:59,124 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:53:59,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:59,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:53:59,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:53:59,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:53:59,124 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:53:59,124 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:53:59,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:53:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:53:59,128 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2ec25b07a2f34a14b0e8943ff59cbde2, entries=200, sequenceid=451, filesize=14.3 K 2023-04-10 02:53:59,130 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/3301f68c06af4ad8895f4fe177e429f7 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/3301f68c06af4ad8895f4fe177e429f7 2023-04-10 02:53:59,135 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/3301f68c06af4ad8895f4fe177e429f7, entries=150, sequenceid=451, filesize=11.9 K 2023-04-10 02:53:59,136 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/0add03d448ce49a4a7f835b71a03dbb1 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/0add03d448ce49a4a7f835b71a03dbb1 2023-04-10 02:53:59,143 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/0add03d448ce49a4a7f835b71a03dbb1, entries=150, sequenceid=451, filesize=11.9 K 2023-04-10 02:53:59,146 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for 66e59774c088f69ab4d8d1711037d595 in 516ms, sequenceid=451, compaction requested=true 2023-04-10 02:53:59,146 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:59,146 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:59,146 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:53:59,147 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:53:59,146 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:53:59,147 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:53:59,149 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63882 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:53:59,149 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/A is initiating minor compaction (all files) 2023-04-10 02:53:59,149 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61442 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:53:59,149 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/A in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:59,149 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/B is initiating minor compaction (all files) 2023-04-10 02:53:59,149 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/B in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:59,149 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/f6340540963648bf85a81d2ffa867cd2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a0c5aaafde274aac88811b17006e30d2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/66e90230189e4c6396622355fb7165ff, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/7215e69ca87244fb92668c9c774de9a5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2ec25b07a2f34a14b0e8943ff59cbde2] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=62.4 K 2023-04-10 02:53:59,149 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/fe031d88692b47deb17111dd2a1c66b6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6016299517cd4523940a8172fbaedcfb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/ab763019e6444286a6666bbc5be47bc2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/9d4d2bd708e742f79431023b97887e6d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/3301f68c06af4ad8895f4fe177e429f7] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=60.0 K 2023-04-10 02:53:59,150 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting fe031d88692b47deb17111dd2a1c66b6, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1681095234752 2023-04-10 02:53:59,150 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting f6340540963648bf85a81d2ffa867cd2, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1681095234752 2023-04-10 02:53:59,152 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6016299517cd4523940a8172fbaedcfb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1681095235900 2023-04-10 02:53:59,153 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting a0c5aaafde274aac88811b17006e30d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1681095235900 2023-04-10 02:53:59,153 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting ab763019e6444286a6666bbc5be47bc2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1681095236241 2023-04-10 02:53:59,153 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 66e90230189e4c6396622355fb7165ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1681095236241 2023-04-10 02:53:59,154 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 9d4d2bd708e742f79431023b97887e6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1681095236880 2023-04-10 02:53:59,154 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 3301f68c06af4ad8895f4fe177e429f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681095238016 2023-04-10 02:53:59,154 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 7215e69ca87244fb92668c9c774de9a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1681095236880 2023-04-10 02:53:59,155 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 2ec25b07a2f34a14b0e8943ff59cbde2, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681095238004 2023-04-10 02:53:59,174 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#A#compaction#60 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:59,182 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#B#compaction#61 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:59,214 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/a78e14b744c4418b954c702dcb55eba6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a78e14b744c4418b954c702dcb55eba6 2023-04-10 02:53:59,221 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/A of 66e59774c088f69ab4d8d1711037d595 into a78e14b744c4418b954c702dcb55eba6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:59,221 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:59,221 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/A, priority=11, startTime=1681095239146; duration=0sec 2023-04-10 02:53:59,221 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:53:59,221 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:53:59,223 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61442 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:53:59,224 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/C is initiating minor compaction (all files) 2023-04-10 02:53:59,224 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/C in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:59,224 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/0bb99e10d5c240998c690fea08d9c473, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/7b427f992bf54ff0acc2448e6461cbff, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/547e4ee6b5164432878370720cf03ee5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/1acfca52020142189f2316a13c85b598, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/0add03d448ce49a4a7f835b71a03dbb1] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=60.0 K 2023-04-10 02:53:59,225 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:53:59,225 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 0bb99e10d5c240998c690fea08d9c473, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1681095234752 2023-04-10 02:53:59,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:53:59,227 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 7b427f992bf54ff0acc2448e6461cbff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1681095235900 2023-04-10 02:53:59,227 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 547e4ee6b5164432878370720cf03ee5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1681095236241 2023-04-10 02:53:59,228 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 1acfca52020142189f2316a13c85b598, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1681095236880 2023-04-10 02:53:59,228 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 0add03d448ce49a4a7f835b71a03dbb1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681095238016 2023-04-10 02:53:59,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:53:59,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:53:59,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:53:59,230 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:53:59,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:59,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:53:59,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:53:59,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:59,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:59,233 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,233 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:53:59,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:53:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:53:59,233 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:59,233 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:53:59,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:53:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:53:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:53:59,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:53:59,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:53:59,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:53:59,236 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:53:59,236 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:53:59,236 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:53:59,244 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:53:59,244 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:53:59,244 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:53:59,244 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:53:59,244 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-10 02:53:59,244 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:53:59,245 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:59,245 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:53:59,245 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:59,245 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:53:59,245 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:53:59,248 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#C#compaction#62 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:53:59,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:53:59,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:53:59,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095299301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095299301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095299305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095299306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095299306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095299407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095299407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095299408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095299408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095299411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095299614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,615 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/df48455438734f068f43216d91611f8e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/df48455438734f068f43216d91611f8e 2023-04-10 02:53:59,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095299615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095299616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095299616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095299617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,624 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/B of 66e59774c088f69ab4d8d1711037d595 into df48455438734f068f43216d91611f8e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:59,624 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:59,624 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/B, priority=11, startTime=1681095239146; duration=0sec 2023-04-10 02:53:59,624 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:59,662 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/a8c95e587fd44aa4a5f4663ab0039355 2023-04-10 02:53:59,668 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/fd68f7ee926849fa8ee82f984c546782 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/fd68f7ee926849fa8ee82f984c546782 2023-04-10 02:53:59,675 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/C of 66e59774c088f69ab4d8d1711037d595 into fd68f7ee926849fa8ee82f984c546782(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:53:59,675 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:53:59,675 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/C, priority=11, startTime=1681095239147; duration=0sec 2023-04-10 02:53:59,676 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:53:59,683 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/5c5d812ed84d4b0a90f8eb8da48c3ec8 2023-04-10 02:53:59,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095299917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095299917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095299919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095299920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:53:59,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:53:59,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095299920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,096 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/fc72ce5325864607ac7342d471cc1f41 2023-04-10 02:54:00,102 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/a8c95e587fd44aa4a5f4663ab0039355 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a8c95e587fd44aa4a5f4663ab0039355 2023-04-10 02:54:00,108 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a8c95e587fd44aa4a5f4663ab0039355, entries=150, sequenceid=472, filesize=11.9 K 2023-04-10 02:54:00,109 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/5c5d812ed84d4b0a90f8eb8da48c3ec8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/5c5d812ed84d4b0a90f8eb8da48c3ec8 2023-04-10 02:54:00,114 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/5c5d812ed84d4b0a90f8eb8da48c3ec8, entries=150, sequenceid=472, filesize=11.9 K 2023-04-10 02:54:00,115 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/fc72ce5325864607ac7342d471cc1f41 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/fc72ce5325864607ac7342d471cc1f41 2023-04-10 02:54:00,120 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/fc72ce5325864607ac7342d471cc1f41, entries=150, sequenceid=472, filesize=11.9 K 2023-04-10 02:54:00,121 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 66e59774c088f69ab4d8d1711037d595 in 877ms, sequenceid=472, compaction requested=false 2023-04-10 02:54:00,121 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:00,121 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:00,121 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:00,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:00,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:00,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:00,128 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,128 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:00,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:00,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bbf8e00[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:00,130 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:00,130 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,132 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:00,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:00,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:00,132 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,132 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:00,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:00,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:00,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:00,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:00,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:00,137 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:00,137 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:00,137 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:00,137 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:00,137 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:00,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:00,139 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,141 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,143 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,143 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,143 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,143 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:00,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:00,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:00,143 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:00,143 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:00,143 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:00,144 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:00,144 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,144 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:00,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:00,144 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,144 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:00,244 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:00,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:00,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:00,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:00,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:00,248 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:00,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:00,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:00,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:00,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,250 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:00,250 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:00,250 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,250 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:00,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:00,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:00,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:00,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:00,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:00,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:00,252 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:00,253 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:00,253 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:54:00,253 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:00,253 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:00,253 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:00,253 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:00,253 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:00,253 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:00,261 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/29b3160e21bc491d8c6186cca8dc8a55 2023-04-10 02:54:00,281 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/7ff95fa517884da8a5af14f807d02d36 2023-04-10 02:54:00,295 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/86c243c8db5e43cbae5f896778f0723d 2023-04-10 02:54:00,304 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/29b3160e21bc491d8c6186cca8dc8a55 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/29b3160e21bc491d8c6186cca8dc8a55 2023-04-10 02:54:00,311 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/29b3160e21bc491d8c6186cca8dc8a55, entries=150, sequenceid=486, filesize=11.9 K 2023-04-10 02:54:00,312 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/7ff95fa517884da8a5af14f807d02d36 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7ff95fa517884da8a5af14f807d02d36 2023-04-10 02:54:00,319 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7ff95fa517884da8a5af14f807d02d36, entries=150, sequenceid=486, filesize=11.9 K 2023-04-10 02:54:00,320 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/86c243c8db5e43cbae5f896778f0723d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/86c243c8db5e43cbae5f896778f0723d 2023-04-10 02:54:00,325 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/86c243c8db5e43cbae5f896778f0723d, entries=150, sequenceid=486, filesize=11.9 K 2023-04-10 02:54:00,326 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 66e59774c088f69ab4d8d1711037d595 in 73ms, sequenceid=486, compaction requested=true 2023-04-10 02:54:00,326 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:00,326 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:00,326 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:00,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:00,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:00,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:00,332 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,332 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,333 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,333 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:00,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f30ba97[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:00,335 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:00,335 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,337 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:00,337 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:00,337 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:00,337 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,337 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:00,338 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:00,338 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,338 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,338 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:00,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:00,341 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:00,341 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:00,341 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:00,341 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:00,341 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:00,343 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,343 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,343 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:00,344 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,346 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,350 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,350 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:00,350 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:00,350 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:00,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:00,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:00,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,351 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:00,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,351 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:00,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:00,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,351 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:00,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:54:00,428 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:54:00,428 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:00,428 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:00,428 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:00,428 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:00,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:00,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:00,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095240435, max timestamp=1681095240446Num uniques 859; 2023-04-10 02:54:00,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:54:00,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095240435, max timestamp=1681095240446Num uniques 859; 2023-04-10 02:54:00,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095240435, max timestamp=1681095240446Num uniques 859; 2023-04-10 02:54:00,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:54:00,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:54:00,452 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:00,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:00,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:00,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:00,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:00,455 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:00,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:00,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:00,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:00,456 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,456 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,457 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:00,457 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,458 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,458 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:00,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:00,459 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:00,459 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:00,459 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:00,460 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:00,460 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:00,460 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:00,460 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:00,460 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:00,460 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:00,460 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:00,460 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:00,460 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,460 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:00,460 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:00,462 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,462 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,462 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,462 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:00,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095300456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095300457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095300458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095300460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19c691e5[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:00,464 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:00,465 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095300461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,466 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,466 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:00,466 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:00,466 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:00,466 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,466 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:00,468 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,468 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:00,469 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:00,469 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:00,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:00,471 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:00,471 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:00,471 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:00,471 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:00,471 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:00,473 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,473 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,473 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:00,474 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,478 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,482 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:00,482 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,482 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:00,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:00,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:00,482 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:00,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:00,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:00,482 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,482 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,482 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:00,482 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:00,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:00,482 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:00,482 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:00,482 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,483 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095300563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095300564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095300564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095300566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095300566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,583 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:00,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:00,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:00,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:00,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:00,586 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:00,586 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:00,586 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:00,586 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:00,587 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,587 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,588 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:00,588 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:00,588 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,588 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:00,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:00,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:00,590 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:00,590 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:00,590 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:00,590 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:00,592 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:00,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e880fe[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:00,594 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:00,594 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,598 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,598 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:00,598 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:00,598 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:00,598 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,598 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:00,600 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:00,602 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:00,602 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:00,603 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:00,603 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:00,603 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:00,606 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,606 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,606 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:00,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,612 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,612 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:00,612 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:00,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:00,612 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:00,612 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,613 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,613 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,613 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,613 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,613 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:00,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:00,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:00,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:00,613 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:00,613 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:00,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:00,713 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:00,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:00,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:00,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:00,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:00,732 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:00,732 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:00,732 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:00,732 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:00,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,734 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:00,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,735 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:00,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:00,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:00,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:00,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:00,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:00,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:00,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:00,748 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:00,748 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:00,748 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:00,748 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:00,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:00,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:00,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:00,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:00,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,751 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,751 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,751 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:00,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e40ca4d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:00,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:00,755 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,757 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:00,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:00,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:00,757 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,757 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:00,759 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,759 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:00,759 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,759 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:00,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:00,770 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:00,770 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:00,770 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:00,770 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:00,770 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:00,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:00,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095300768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095300768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095300769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095300770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:00,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095300772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,778 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,780 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:00,794 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:00,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:00,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:00,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:00,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:00,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:00,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,795 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,795 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,795 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:00,795 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:00,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:00,842 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/2d7e36f6d7c64f06bc5e248a59cb75dc 2023-04-10 02:54:00,895 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:00,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:00,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:00,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:00,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:00,898 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:00,898 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:00,898 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:00,898 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:00,899 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,899 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,900 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:00,900 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:00,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,901 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,901 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:00,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:00,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:00,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:00,904 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:00,904 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:00,904 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:00,905 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:00,905 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:00,905 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:00,905 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:00,905 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:00,905 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:00,905 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:00,905 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:00,905 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,905 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:00,905 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:00,906 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,907 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24be2b10[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:00,909 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:00,909 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,910 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,910 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:00,911 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,911 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:00,911 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:00,911 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:00,913 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,913 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:00,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:00,917 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:00,917 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:00,917 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:00,917 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:00,917 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:00,918 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,918 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:00,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,918 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:00,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,920 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,921 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,924 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,924 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,924 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:00,924 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:00,924 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:00,924 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,924 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:00,924 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:00,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:00,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:00,924 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:00,924 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:00,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:00,924 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:00,924 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:00,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:01,025 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:01,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:01,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:01,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:01,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:01,029 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:01,029 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:01,029 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:01,029 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:01,029 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,029 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,031 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:01,031 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:01,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:01,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:01,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:01,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:01,032 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:01,032 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:01,032 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:01,032 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:01,032 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:01,033 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:01,033 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:01,033 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:01,033 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:01,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:01,036 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,036 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:01,037 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:01,037 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:01,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:01,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9dadfdb[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:01,038 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:01,038 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,039 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:01,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:01,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:01,040 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,040 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:01,043 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:01,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:01,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:01,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:01,046 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:01,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:01,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:01,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:01,046 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:01,053 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,053 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:01,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,053 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:01,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,054 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:01,056 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:01,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:01,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:01,059 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:01,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:01,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:01,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:01,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,059 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:01,060 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:01,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:01,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095301078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095301078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095301079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095301080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095301080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,160 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:01,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:01,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:01,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:01,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:01,163 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:01,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:01,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:01,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:01,164 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,164 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,165 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:01,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:01,165 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,166 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:01,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:01,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:01,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:01,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:01,167 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:01,167 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:01,167 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:01,167 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:01,169 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,169 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:01,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:01,169 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,169 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:01,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:01,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:01,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1428416e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:01,171 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:01,171 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,177 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,177 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:01,177 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:01,177 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:01,177 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,177 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:01,179 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,179 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:01,179 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:01,179 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:01,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:01,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:01,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:01,182 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:01,182 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:01,182 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:01,182 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:01,182 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:01,183 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,183 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:01,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:01,184 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:01,185 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:01,187 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,190 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:01,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:01,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:01,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:01,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:01,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:01,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,190 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:01,190 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:01,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:01,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:01,259 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/7e385518113443038b165a11da9fcd2c 2023-04-10 02:54:01,274 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/f8cac114940541ee9fc0f868a171393b 2023-04-10 02:54:01,281 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/2d7e36f6d7c64f06bc5e248a59cb75dc as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2d7e36f6d7c64f06bc5e248a59cb75dc 2023-04-10 02:54:01,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2d7e36f6d7c64f06bc5e248a59cb75dc, entries=250, sequenceid=496, filesize=16.7 K 2023-04-10 02:54:01,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/7e385518113443038b165a11da9fcd2c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7e385518113443038b165a11da9fcd2c 2023-04-10 02:54:01,290 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:01,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:01,294 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7e385518113443038b165a11da9fcd2c, entries=150, sequenceid=496, filesize=11.9 K 2023-04-10 02:54:01,295 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/f8cac114940541ee9fc0f868a171393b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/f8cac114940541ee9fc0f868a171393b 2023-04-10 02:54:01,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:01,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:01,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:01,302 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:01,302 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:01,302 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:01,302 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:01,303 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,303 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,304 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,304 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:01,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:01,305 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,305 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:01,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/f8cac114940541ee9fc0f868a171393b, entries=150, sequenceid=496, filesize=11.9 K 2023-04-10 02:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:01,306 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 66e59774c088f69ab4d8d1711037d595 in 878ms, sequenceid=496, compaction requested=true 2023-04-10 02:54:01,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:01,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:01,307 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:54:01,307 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:54:01,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:01,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:01,309 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:54:01,309 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/B is initiating minor compaction (all files) 2023-04-10 02:54:01,309 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/B in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:01,309 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/df48455438734f068f43216d91611f8e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/5c5d812ed84d4b0a90f8eb8da48c3ec8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7ff95fa517884da8a5af14f807d02d36, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7e385518113443038b165a11da9fcd2c] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=48.3 K 2023-04-10 02:54:01,310 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting df48455438734f068f43216d91611f8e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681095238016 2023-04-10 02:54:01,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:01,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:01,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:01,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:01,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:01,310 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 5c5d812ed84d4b0a90f8eb8da48c3ec8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1681095238673 2023-04-10 02:54:01,310 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54306 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:54:01,311 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/A is initiating minor compaction (all files) 2023-04-10 02:54:01,311 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 7ff95fa517884da8a5af14f807d02d36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681095239297 2023-04-10 02:54:01,311 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/A in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:01,311 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a78e14b744c4418b954c702dcb55eba6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a8c95e587fd44aa4a5f4663ab0039355, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/29b3160e21bc491d8c6186cca8dc8a55, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2d7e36f6d7c64f06bc5e248a59cb75dc] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=53.0 K 2023-04-10 02:54:01,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:01,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:01,311 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:01,311 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:01,311 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-10 02:54:01,312 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:01,312 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:01,312 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:01,312 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting a78e14b744c4418b954c702dcb55eba6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681095238016 2023-04-10 02:54:01,312 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:01,312 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 7e385518113443038b165a11da9fcd2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1681095240425 2023-04-10 02:54:01,312 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:01,312 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:01,312 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting a8c95e587fd44aa4a5f4663ab0039355, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1681095238673 2023-04-10 02:54:01,317 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 29b3160e21bc491d8c6186cca8dc8a55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681095239297 2023-04-10 02:54:01,317 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 2d7e36f6d7c64f06bc5e248a59cb75dc, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1681095240424 2023-04-10 02:54:01,336 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#A#compaction#73 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:01,338 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#B#compaction#74 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:01,348 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/309ea6d8d8cb41f6b63e96986ebd1721 2023-04-10 02:54:01,365 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/560bc97bb6914e2db48edf66824ba8b8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/560bc97bb6914e2db48edf66824ba8b8 2023-04-10 02:54:01,372 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/B of 66e59774c088f69ab4d8d1711037d595 into 560bc97bb6914e2db48edf66824ba8b8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:01,373 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:01,373 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/B, priority=12, startTime=1681095241307; duration=0sec 2023-04-10 02:54:01,373 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:01,373 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:54:01,377 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:54:01,377 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/C is initiating minor compaction (all files) 2023-04-10 02:54:01,377 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/C in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:01,378 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/fd68f7ee926849fa8ee82f984c546782, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/fc72ce5325864607ac7342d471cc1f41, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/86c243c8db5e43cbae5f896778f0723d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/f8cac114940541ee9fc0f868a171393b] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=48.3 K 2023-04-10 02:54:01,381 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting fd68f7ee926849fa8ee82f984c546782, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681095238016 2023-04-10 02:54:01,384 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting fc72ce5325864607ac7342d471cc1f41, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1681095238673 2023-04-10 02:54:01,385 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 86c243c8db5e43cbae5f896778f0723d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681095239297 2023-04-10 02:54:01,386 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting f8cac114940541ee9fc0f868a171393b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1681095240425 2023-04-10 02:54:01,388 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/525dc1c10b1c453da2c19643fca20116 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/525dc1c10b1c453da2c19643fca20116 2023-04-10 02:54:01,397 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/A of 66e59774c088f69ab4d8d1711037d595 into 525dc1c10b1c453da2c19643fca20116(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:01,397 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:01,397 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/A, priority=12, startTime=1681095241307; duration=0sec 2023-04-10 02:54:01,397 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:01,405 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#C#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:01,436 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/8fd14dc152f5482e82f72e020dba203d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/8fd14dc152f5482e82f72e020dba203d 2023-04-10 02:54:01,447 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/C of 66e59774c088f69ab4d8d1711037d595 into 8fd14dc152f5482e82f72e020dba203d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:01,447 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:01,447 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/C, priority=12, startTime=1681095241307; duration=0sec 2023-04-10 02:54:01,447 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:01,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:54:01,586 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:01,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095301589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095301590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095301590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095301591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095301591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095301693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095301693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095301693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095301694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,791 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/c75a3d13a2ad4403a17db41ce71afa2b 2023-04-10 02:54:01,803 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/de8c74b06c7a4733bd85a3f0343d259e 2023-04-10 02:54:01,812 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/309ea6d8d8cb41f6b63e96986ebd1721 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/309ea6d8d8cb41f6b63e96986ebd1721 2023-04-10 02:54:01,818 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/309ea6d8d8cb41f6b63e96986ebd1721, entries=150, sequenceid=531, filesize=11.9 K 2023-04-10 02:54:01,819 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/c75a3d13a2ad4403a17db41ce71afa2b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/c75a3d13a2ad4403a17db41ce71afa2b 2023-04-10 02:54:01,825 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/c75a3d13a2ad4403a17db41ce71afa2b, entries=150, sequenceid=531, filesize=11.9 K 2023-04-10 02:54:01,826 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/de8c74b06c7a4733bd85a3f0343d259e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/de8c74b06c7a4733bd85a3f0343d259e 2023-04-10 02:54:01,831 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/de8c74b06c7a4733bd85a3f0343d259e, entries=150, sequenceid=531, filesize=11.9 K 2023-04-10 02:54:01,832 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 66e59774c088f69ab4d8d1711037d595 in 521ms, sequenceid=531, compaction requested=false 2023-04-10 02:54:01,833 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:01,833 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:01,833 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:01,833 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:01,833 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,833 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:01,833 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:01,835 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,835 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:01,835 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,835 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:01,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:01,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c02f304[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:01,839 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:01,839 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,846 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:01,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:01,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:01,847 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,848 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:01,849 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:01,849 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:01,849 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:01,849 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:01,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:01,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:01,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:01,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:01,852 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:01,852 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:01,852 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:01,852 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:01,852 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:01,854 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,854 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,854 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:01,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:01,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:01,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:01,855 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:01,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:01,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:01,856 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:01,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,859 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,859 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,859 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,859 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,860 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,860 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,860 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,860 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:01,860 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:01,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:01,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:01,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:01,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:01,860 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:01,860 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:01,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:01,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:01,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:01,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:54:01,898 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:54:01,900 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:01,900 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:01,900 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:01,900 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:01,900 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:01,900 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:01,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095301913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095301914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095301918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:01,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095301919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,960 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:01,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:01,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:01,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:01,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:01,964 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:01,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:01,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:01,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:01,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,966 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:01,966 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:01,966 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,966 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:01,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:01,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:01,968 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:01,968 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:01,968 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:01,968 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:01,970 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,970 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:01,970 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,970 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:01,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:01,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7931bb3f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:01,972 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:01,972 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,975 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,975 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:01,975 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,975 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:01,975 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:01,975 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:01,977 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,977 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:01,977 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:01,977 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:01,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:01,981 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:01,981 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:01,981 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:01,981 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:01,981 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:01,983 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,983 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:01,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,983 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:01,984 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,985 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:01,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:01,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:01,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:01,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:01,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:01,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:01,991 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:01,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:01,992 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:01,992 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:01,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:01,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:01,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:02,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:02,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095302020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:02,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095302021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:02,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:02,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095302022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095302022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,092 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:02,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:02,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:02,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:02,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:02,095 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:02,095 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:02,095 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:02,096 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:02,096 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,096 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,097 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:02,097 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:02,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:02,098 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,098 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:02,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:02,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:02,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:02,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:02,099 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:02,100 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,100 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:02,100 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:02,100 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:02,103 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,103 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,103 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,103 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:02,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:02,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26ca87c5[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:02,106 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:02,106 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,107 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,107 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:02,107 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,107 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:02,107 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:02,107 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:02,109 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:02,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:02,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:02,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:02,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:02,111 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:02,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:02,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:02,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:02,111 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:02,113 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,113 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,113 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:02,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,114 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,115 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,123 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,123 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,123 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,123 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,123 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,123 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:02,123 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:02,123 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:02,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:02,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:02,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:02,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:02,123 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:02,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,124 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:02,124 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:02,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:02,224 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:02,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:02,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095302224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:02,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095302224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:02,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:02,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:02,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095302227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:02,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095302227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:02,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:02,229 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:02,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:02,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:02,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:02,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,231 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:02,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:02,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:02,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:02,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:02,233 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,233 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:02,233 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:02,233 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:02,235 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,235 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,235 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,235 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:02,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70bf9143[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:02,237 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:02,237 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,238 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:02,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:02,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:02,238 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,238 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:02,240 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,240 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:02,241 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:02,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,241 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:02,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:02,244 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:02,244 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:02,244 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:02,244 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:02,244 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:02,245 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,246 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:02,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,246 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,246 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,250 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,250 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:02,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:02,250 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,250 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:02,250 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:02,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:02,251 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:02,251 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:02,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:02,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:02,250 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:02,250 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:02,311 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/92b42e0b61e94489ba411c9dcbc38c11 2023-04-10 02:54:02,351 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:02,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:02,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:02,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:02,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:02,355 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:02,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:02,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:02,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:02,356 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,356 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,357 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:02,357 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:02,357 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,357 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:02,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:02,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:02,359 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,359 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:02,359 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:02,359 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:02,362 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,362 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,363 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,363 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:02,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1ed683[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:02,365 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:02,365 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,366 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:02,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:02,367 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,367 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:02,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:02,368 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,369 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:02,369 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:02,369 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:02,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:02,371 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:02,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:02,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:02,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:02,371 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:02,373 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,373 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,373 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,373 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:02,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:02,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,375 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:02,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:02,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:02,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:02,382 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,382 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:02,382 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:02,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:02,382 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,383 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:02,383 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:02,483 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:02,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:02,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:02,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:02,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:02,487 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:02,487 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:02,487 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:02,487 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:02,487 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,487 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,497 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:02,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:02,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:02,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:02,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:02,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:02,499 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:02,499 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:02,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:02,509 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:02,509 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:02,510 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,510 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:02,510 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:02,510 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,510 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:02,510 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:02,510 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,510 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:02,510 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:02,512 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,512 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,513 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,513 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:02,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f82749c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:02,514 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:02,514 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,515 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:02,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:02,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:02,516 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,516 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:02,517 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:02,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:02,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:02,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:02,520 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:02,520 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:02,520 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:02,520 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:02,520 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:02,522 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,522 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,522 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:02,522 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:02,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,524 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:02,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095302526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,527 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:02,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:02,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:02,527 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:02,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:02,527 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,527 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:02,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:02,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:02,529 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:02,529 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:02,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:02,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095302529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:02,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095302530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:02,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095302530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:02,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095302602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,630 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:02,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:02,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:02,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:02,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:02,638 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:02,639 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:02,639 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:02,639 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:02,639 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,639 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,640 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,641 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:02,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:02,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:02,641 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,641 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:02,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:02,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:02,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:02,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:02,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:02,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:02,643 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:02,643 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:02,643 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,643 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:02,643 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:02,643 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,643 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:02,643 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:02,643 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,643 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:02,643 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:02,645 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,645 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,645 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,645 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:02,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d042806[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:02,647 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:02,647 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,652 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,652 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:02,652 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:02,652 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:02,652 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,652 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:02,654 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:02,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:02,656 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:02,657 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:02,657 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:02,657 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:02,657 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:02,659 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,659 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,659 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:02,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,659 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:02,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:02,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,661 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,663 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:02,663 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:02,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:02,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:02,664 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,664 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,664 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:02,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:02,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:02,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:02,664 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,664 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,664 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,664 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:02,664 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:02,664 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,725 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/af7710fab0c046e5b2e4128608f1bce9 2023-04-10 02:54:02,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/11b1aec82f724c6aa2a4bfab6ca14e4d 2023-04-10 02:54:02,753 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/92b42e0b61e94489ba411c9dcbc38c11 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/92b42e0b61e94489ba411c9dcbc38c11 2023-04-10 02:54:02,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/92b42e0b61e94489ba411c9dcbc38c11, entries=150, sequenceid=548, filesize=11.9 K 2023-04-10 02:54:02,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/af7710fab0c046e5b2e4128608f1bce9 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/af7710fab0c046e5b2e4128608f1bce9 2023-04-10 02:54:02,764 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:02,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:02,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/af7710fab0c046e5b2e4128608f1bce9, entries=150, sequenceid=548, filesize=11.9 K 2023-04-10 02:54:02,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:02,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/11b1aec82f724c6aa2a4bfab6ca14e4d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/11b1aec82f724c6aa2a4bfab6ca14e4d 2023-04-10 02:54:02,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:02,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:02,769 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:02,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:02,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:02,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:02,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,771 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:02,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:02,772 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,772 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,772 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,773 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/11b1aec82f724c6aa2a4bfab6ca14e4d, entries=150, sequenceid=548, filesize=11.9 K 2023-04-10 02:54:02,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:02,774 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 66e59774c088f69ab4d8d1711037d595 in 876ms, sequenceid=548, compaction requested=true 2023-04-10 02:54:02,774 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:02,775 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:02,775 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:02,775 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:02,775 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:02,775 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:02,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:02,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:02,775 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:02,775 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:02,775 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:02,776 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:02,776 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:02,776 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,776 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:02,776 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-10 02:54:02,776 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:02,776 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:02,776 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/A is initiating minor compaction (all files) 2023-04-10 02:54:02,776 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:02,776 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/A in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,776 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/B is initiating minor compaction (all files) 2023-04-10 02:54:02,776 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:02,777 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/525dc1c10b1c453da2c19643fca20116, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/309ea6d8d8cb41f6b63e96986ebd1721, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/92b42e0b61e94489ba411c9dcbc38c11] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=36.5 K 2023-04-10 02:54:02,777 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/B in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,777 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:02,777 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/560bc97bb6914e2db48edf66824ba8b8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/c75a3d13a2ad4403a17db41ce71afa2b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/af7710fab0c046e5b2e4128608f1bce9] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=36.5 K 2023-04-10 02:54:02,777 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:02,777 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:02,777 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:02,777 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 525dc1c10b1c453da2c19643fca20116, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1681095240425 2023-04-10 02:54:02,777 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 560bc97bb6914e2db48edf66824ba8b8, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1681095240425 2023-04-10 02:54:02,778 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 309ea6d8d8cb41f6b63e96986ebd1721, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1681095240455 2023-04-10 02:54:02,778 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting c75a3d13a2ad4403a17db41ce71afa2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1681095240455 2023-04-10 02:54:02,779 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 92b42e0b61e94489ba411c9dcbc38c11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681095241897 2023-04-10 02:54:02,779 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting af7710fab0c046e5b2e4128608f1bce9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681095241897 2023-04-10 02:54:02,793 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#B#compaction#82 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:02,793 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#A#compaction#83 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:02,805 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/6fb4240684c248f1b7b4328aca8f70c5 2023-04-10 02:54:02,841 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/7837ee0803014440bf0dfa6c9ae988fd as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7837ee0803014440bf0dfa6c9ae988fd 2023-04-10 02:54:02,842 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/00feab6c288d4ff59efe72c924c8c084 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/00feab6c288d4ff59efe72c924c8c084 2023-04-10 02:54:02,845 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/5c844cd309c24bfcaed4a05a4562e84f 2023-04-10 02:54:02,849 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/A of 66e59774c088f69ab4d8d1711037d595 into 00feab6c288d4ff59efe72c924c8c084(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:02,849 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:02,849 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/A, priority=13, startTime=1681095242775; duration=0sec 2023-04-10 02:54:02,849 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:02,849 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:02,851 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:02,851 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/C is initiating minor compaction (all files) 2023-04-10 02:54:02,851 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/C in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,851 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/8fd14dc152f5482e82f72e020dba203d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/de8c74b06c7a4733bd85a3f0343d259e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/11b1aec82f724c6aa2a4bfab6ca14e4d] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=36.5 K 2023-04-10 02:54:02,852 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 8fd14dc152f5482e82f72e020dba203d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1681095240425 2023-04-10 02:54:02,852 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/B of 66e59774c088f69ab4d8d1711037d595 into 7837ee0803014440bf0dfa6c9ae988fd(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:02,852 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:02,852 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/B, priority=13, startTime=1681095242775; duration=0sec 2023-04-10 02:54:02,853 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting de8c74b06c7a4733bd85a3f0343d259e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1681095240455 2023-04-10 02:54:02,853 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:02,853 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 11b1aec82f724c6aa2a4bfab6ca14e4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681095241897 2023-04-10 02:54:02,862 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#C#compaction#86 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:02,871 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/5b9243b21d3c4e87b7e8efdf4e26b382 2023-04-10 02:54:02,877 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/6fb4240684c248f1b7b4328aca8f70c5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/6fb4240684c248f1b7b4328aca8f70c5 2023-04-10 02:54:02,890 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/6fb4240684c248f1b7b4328aca8f70c5, entries=150, sequenceid=566, filesize=11.9 K 2023-04-10 02:54:02,891 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/5c844cd309c24bfcaed4a05a4562e84f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/5c844cd309c24bfcaed4a05a4562e84f 2023-04-10 02:54:02,898 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/86dc333fa1f84305a91a4f5c60426410 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/86dc333fa1f84305a91a4f5c60426410 2023-04-10 02:54:02,903 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/5c844cd309c24bfcaed4a05a4562e84f, entries=150, sequenceid=566, filesize=11.9 K 2023-04-10 02:54:02,904 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/5b9243b21d3c4e87b7e8efdf4e26b382 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/5b9243b21d3c4e87b7e8efdf4e26b382 2023-04-10 02:54:02,906 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/C of 66e59774c088f69ab4d8d1711037d595 into 86dc333fa1f84305a91a4f5c60426410(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:02,906 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:02,906 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/C, priority=13, startTime=1681095242775; duration=0sec 2023-04-10 02:54:02,907 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:02,910 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/5b9243b21d3c4e87b7e8efdf4e26b382, entries=150, sequenceid=566, filesize=11.9 K 2023-04-10 02:54:02,910 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 66e59774c088f69ab4d8d1711037d595 in 134ms, sequenceid=566, compaction requested=false 2023-04-10 02:54:02,911 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:02,911 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:02,911 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:02,911 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:02,911 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,911 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:02,911 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:02,913 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,913 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,913 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,913 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:02,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41bd9d6c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:02,916 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:02,916 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,917 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,917 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:02,917 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:02,917 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:02,917 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,918 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:02,919 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,920 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:02,920 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:02,920 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:02,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:02,923 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:02,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:02,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:02,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:02,923 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:02,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:02,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:02,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:02,927 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:02,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:02,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:02,931 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:02,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:02,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:02,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:02,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:02,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:02,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:02,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:02,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:02,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:02,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:02,932 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:02,932 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:03,032 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:03,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:03,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:03,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:03,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:03,046 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:03,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:03,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:03,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:03,047 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,047 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:54:03,049 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:54:03,050 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:03,050 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:03,051 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:03,051 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:03,051 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:03,051 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:03,051 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:03,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:03,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:03,051 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,052 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,052 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:03,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:03,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:03,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:03,064 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:03,064 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:03,064 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:03,064 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,065 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:03,065 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,069 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:03,070 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,070 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:03,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:03,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7caae116[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:03,072 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:03,072 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,073 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:03,076 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,076 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:03,078 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:03,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681095243066, max timestamp=1681095243075Num uniques 859; 2023-04-10 02:54:03,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-10 02:54:03,078 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:03,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681095243066, max timestamp=1681095243075Num uniques 859; 2023-04-10 02:54:03,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-10 02:54:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681095243066, max timestamp=1681095243075Num uniques 859; 2023-04-10 02:54:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:03,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-10 02:54:03,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:03,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:03,083 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:03,084 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:03,084 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:03,084 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:03,084 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:03,086 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/e3427564eb7b4048aa99475c9ad4b8dd 2023-04-10 02:54:03,088 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,088 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,088 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:03,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:03,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:03,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,090 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,095 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,095 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,095 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,095 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,095 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:03,095 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:03,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:03,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:03,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:03,095 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,096 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:03,096 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:03,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:03,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:03,095 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:03,096 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,096 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:03,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:03,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095303096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095303097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095303098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095303099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,133 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/78b1495d46c74c509de912e43fc9aaa5 2023-04-10 02:54:03,196 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:03,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:03,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:03,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:03,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:03,199 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:03,199 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:03,199 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:03,199 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:03,200 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,200 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095303201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095303201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095303203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,206 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,206 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:03,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:03,206 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,206 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095303204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:03,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:03,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:03,207 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:03,207 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:03,207 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:03,208 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:03,208 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:03,208 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:03,208 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:03,210 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:03,210 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,210 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:03,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:03,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a05c8b[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:03,212 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:03,212 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,213 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,213 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:03,213 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:03,213 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:03,214 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,214 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:03,216 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,216 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:03,216 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:03,216 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:03,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:03,219 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:03,219 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:03,219 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:03,219 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:03,219 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:03,221 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,221 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:03,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:03,222 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,223 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,226 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,226 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,226 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,226 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,226 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,226 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:03,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:03,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:03,226 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:03,226 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:03,226 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:03,227 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:03,227 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:03,227 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:03,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:03,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:03,327 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:03,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:03,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:03,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:03,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:03,360 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:03,360 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:03,360 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:03,360 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:03,360 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,361 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,362 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,362 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:03,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:03,363 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,363 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:03,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:03,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:03,364 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:03,364 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:03,364 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:03,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:03,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:03,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:03,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:03,376 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:03,376 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:03,379 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,380 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:03,381 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,381 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:03,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:03,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c880a81[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:03,381 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:03,381 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,383 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,383 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:03,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,383 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:03,383 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:03,383 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:03,385 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:03,388 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:03,388 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:03,388 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:03,388 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:03,388 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:03,390 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,390 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:03,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,390 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:03,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:03,390 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,391 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,395 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,395 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,395 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:03,395 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:03,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:03,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:03,395 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:03,395 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:03,395 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,395 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:03,396 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:03,395 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,396 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,396 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:03,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:03,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095303406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095303406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095303406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095303408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,496 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:03,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:03,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:03,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:03,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:03,500 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:03,500 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:03,500 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:03,500 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:03,501 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,501 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,502 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,502 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:03,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:03,502 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,502 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:03,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:03,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:03,504 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:03,504 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:03,504 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:03,508 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:03,508 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:03,508 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:03,508 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:03,510 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,510 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:03,510 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,510 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:03,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:03,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11f3a681[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:03,512 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:03,512 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,516 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:03,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:03,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:03,516 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,517 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:03,518 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:03,518 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:03,518 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:03,518 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:03,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:03,520 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:03,520 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:03,520 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:03,521 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:03,521 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:03,522 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,522 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:03,522 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:03,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:03,523 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,524 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:03,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:03,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:03,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:03,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:03,528 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:03,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:03,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:03,528 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:03,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:03,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:03,529 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:03,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/27a6ce81a1984088a057622bc24cb82c 2023-04-10 02:54:03,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/e3427564eb7b4048aa99475c9ad4b8dd as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/e3427564eb7b4048aa99475c9ad4b8dd 2023-04-10 02:54:03,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/e3427564eb7b4048aa99475c9ad4b8dd, entries=150, sequenceid=580, filesize=11.9 K 2023-04-10 02:54:03,586 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/78b1495d46c74c509de912e43fc9aaa5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/78b1495d46c74c509de912e43fc9aaa5 2023-04-10 02:54:03,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/78b1495d46c74c509de912e43fc9aaa5, entries=150, sequenceid=580, filesize=11.9 K 2023-04-10 02:54:03,592 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/27a6ce81a1984088a057622bc24cb82c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/27a6ce81a1984088a057622bc24cb82c 2023-04-10 02:54:03,597 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/27a6ce81a1984088a057622bc24cb82c, entries=150, sequenceid=580, filesize=11.9 K 2023-04-10 02:54:03,598 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 66e59774c088f69ab4d8d1711037d595 in 549ms, sequenceid=580, compaction requested=true 2023-04-10 02:54:03,598 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:03,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:03,598 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:03,598 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:03,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:03,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:03,599 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:03,599 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/A is initiating minor compaction (all files) 2023-04-10 02:54:03,599 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/A in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:03,600 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/00feab6c288d4ff59efe72c924c8c084, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/6fb4240684c248f1b7b4328aca8f70c5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/e3427564eb7b4048aa99475c9ad4b8dd] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=36.6 K 2023-04-10 02:54:03,600 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:03,600 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/B is initiating minor compaction (all files) 2023-04-10 02:54:03,600 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/B in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:03,600 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 00feab6c288d4ff59efe72c924c8c084, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681095241897 2023-04-10 02:54:03,600 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7837ee0803014440bf0dfa6c9ae988fd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/5c844cd309c24bfcaed4a05a4562e84f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/78b1495d46c74c509de912e43fc9aaa5] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=36.6 K 2023-04-10 02:54:03,601 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 6fb4240684c248f1b7b4328aca8f70c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1681095241908 2023-04-10 02:54:03,601 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 7837ee0803014440bf0dfa6c9ae988fd, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681095241897 2023-04-10 02:54:03,601 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting e3427564eb7b4048aa99475c9ad4b8dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681095243040 2023-04-10 02:54:03,601 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 5c844cd309c24bfcaed4a05a4562e84f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1681095241908 2023-04-10 02:54:03,602 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 78b1495d46c74c509de912e43fc9aaa5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681095243040 2023-04-10 02:54:03,615 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#A#compaction#90 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:03,618 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#B#compaction#91 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:03,629 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:03,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:03,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:03,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:03,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:03,638 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:03,638 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:03,638 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:03,638 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:03,639 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,639 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:03,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:03,641 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,641 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,641 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:03,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:03,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:03,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:03,647 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:03,647 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:03,647 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:03,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:03,648 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:03,648 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:03,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:03,648 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-10 02:54:03,648 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:03,648 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:03,649 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:03,649 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:03,649 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:03,649 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:03,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:54:03,711 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:03,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095303718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095303718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095303718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095303719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095303825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095303825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095303825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:03,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:03,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095303825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:04,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095304026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:04,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095304029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:04,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095304029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:04,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095304029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,035 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/3ac6b7b7da7848a299866825d94eea48 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3ac6b7b7da7848a299866825d94eea48 2023-04-10 02:54:04,041 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/A of 66e59774c088f69ab4d8d1711037d595 into 3ac6b7b7da7848a299866825d94eea48(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:04,041 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:04,041 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/A, priority=13, startTime=1681095243598; duration=0sec 2023-04-10 02:54:04,041 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:04,041 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:04,042 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:04,042 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/C is initiating minor compaction (all files) 2023-04-10 02:54:04,043 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/C in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:04,043 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/86dc333fa1f84305a91a4f5c60426410, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/5b9243b21d3c4e87b7e8efdf4e26b382, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/27a6ce81a1984088a057622bc24cb82c] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=36.6 K 2023-04-10 02:54:04,043 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 86dc333fa1f84305a91a4f5c60426410, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681095241897 2023-04-10 02:54:04,044 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 5b9243b21d3c4e87b7e8efdf4e26b382, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1681095241908 2023-04-10 02:54:04,044 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 27a6ce81a1984088a057622bc24cb82c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681095243040 2023-04-10 02:54:04,046 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/39cf26f3fae847bb9383854ad668f7ec as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/39cf26f3fae847bb9383854ad668f7ec 2023-04-10 02:54:04,053 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#C#compaction#93 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:04,053 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/B of 66e59774c088f69ab4d8d1711037d595 into 39cf26f3fae847bb9383854ad668f7ec(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:04,053 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:04,053 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/B, priority=13, startTime=1681095243598; duration=0sec 2023-04-10 02:54:04,053 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:04,062 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/614fa69df5f143a394b683b97bd4849a 2023-04-10 02:54:04,068 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/844c503a66ff431db13ed21e657c7e5f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/844c503a66ff431db13ed21e657c7e5f 2023-04-10 02:54:04,076 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/C of 66e59774c088f69ab4d8d1711037d595 into 844c503a66ff431db13ed21e657c7e5f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:04,076 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:04,076 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/C, priority=13, startTime=1681095243598; duration=0sec 2023-04-10 02:54:04,080 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:04,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:04,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095304331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:04,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095304332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:04,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095304333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:04,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095304333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,486 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/72fbf10769b6420d8ef667b7b9b0f5fd 2023-04-10 02:54:04,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:04,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095304612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,616 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:54:04,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:04,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095304836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:04,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095304837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:04,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095304837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:04,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095304838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,910 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/de40f81f673b465d92b5d8cfae4494f5 2023-04-10 02:54:04,916 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/614fa69df5f143a394b683b97bd4849a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/614fa69df5f143a394b683b97bd4849a 2023-04-10 02:54:04,922 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/614fa69df5f143a394b683b97bd4849a, entries=150, sequenceid=614, filesize=11.9 K 2023-04-10 02:54:04,923 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/72fbf10769b6420d8ef667b7b9b0f5fd as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/72fbf10769b6420d8ef667b7b9b0f5fd 2023-04-10 02:54:04,929 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/72fbf10769b6420d8ef667b7b9b0f5fd, entries=150, sequenceid=614, filesize=11.9 K 2023-04-10 02:54:04,934 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/de40f81f673b465d92b5d8cfae4494f5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/de40f81f673b465d92b5d8cfae4494f5 2023-04-10 02:54:04,940 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/de40f81f673b465d92b5d8cfae4494f5, entries=150, sequenceid=614, filesize=11.9 K 2023-04-10 02:54:04,941 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 66e59774c088f69ab4d8d1711037d595 in 1293ms, sequenceid=614, compaction requested=false 2023-04-10 02:54:04,941 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:04,941 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:04,941 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:04,941 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:04,941 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,943 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:04,943 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:04,945 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:04,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:04,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:04,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:04,947 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:04,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a7a57f8[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:04,947 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:04,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:04,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:04,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:04,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:04,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:04,948 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,948 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:04,950 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:04,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:04,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:04,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:04,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:04,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:04,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:04,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:04,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:04,953 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:04,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:04,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:04,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:04,953 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:04,955 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:04,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:04,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:04,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:04,956 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:04,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:04,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:04,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:04,957 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:04,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:04,960 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:04,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:04,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:04,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:04,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:04,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:04,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:04,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:04,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:04,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:04,960 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:04,961 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:04,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:04,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:04,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:05,061 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:05,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:05,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:05,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:05,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:05,066 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:05,066 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:05,066 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:05,066 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:05,067 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,067 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,070 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,070 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:05,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:05,070 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,070 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:05,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:05,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:05,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:05,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:05,073 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:05,073 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:05,073 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:05,073 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:05,073 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:05,073 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:05,073 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:05,074 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:54:05,074 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:05,074 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:05,074 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:05,074 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:05,074 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:05,075 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:05,496 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=628 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/07725401a83b43b3b25199208f7c31cb 2023-04-10 02:54:05,521 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=628 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/1866d6ee9a0642139b0c74609c95095b 2023-04-10 02:54:05,536 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=628 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/d8266c5a26c0492ba602f3f00583c044 2023-04-10 02:54:05,542 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/07725401a83b43b3b25199208f7c31cb as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/07725401a83b43b3b25199208f7c31cb 2023-04-10 02:54:05,552 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/07725401a83b43b3b25199208f7c31cb, entries=100, sequenceid=628, filesize=9.5 K 2023-04-10 02:54:05,553 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/1866d6ee9a0642139b0c74609c95095b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/1866d6ee9a0642139b0c74609c95095b 2023-04-10 02:54:05,558 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/1866d6ee9a0642139b0c74609c95095b, entries=100, sequenceid=628, filesize=9.5 K 2023-04-10 02:54:05,559 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/d8266c5a26c0492ba602f3f00583c044 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/d8266c5a26c0492ba602f3f00583c044 2023-04-10 02:54:05,564 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/d8266c5a26c0492ba602f3f00583c044, entries=100, sequenceid=628, filesize=9.5 K 2023-04-10 02:54:05,565 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 66e59774c088f69ab4d8d1711037d595 in 492ms, sequenceid=628, compaction requested=true 2023-04-10 02:54:05,565 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:05,565 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:05,565 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:05,565 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:05,565 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,566 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:05,566 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:05,567 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,567 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:05,567 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,568 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:05,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:05,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@686c1038[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:05,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:05,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,571 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,571 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:05,571 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:05,571 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:05,571 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,571 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:05,573 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,573 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:05,573 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:05,573 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:05,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:05,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:05,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:05,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:05,576 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:05,576 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:05,576 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:05,576 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:05,576 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:05,578 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,578 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:05,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,578 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:05,579 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,580 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:05,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,583 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,583 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,583 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,583 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,583 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,583 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,583 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:05,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:05,583 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:05,583 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:05,583 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:05,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:05,583 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:05,583 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:05,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:05,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:05,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:05,694 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:05,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:05,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:05,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:05,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:05,702 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:05,702 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:05,702 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:05,702 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:05,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,706 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,706 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:05,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:05,707 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,707 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:05,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:05,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:05,713 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:05,713 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:05,713 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:05,722 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:05,722 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:05,722 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:05,722 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:05,723 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:05,723 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:05,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:05,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:05,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:05,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:05,725 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,725 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:05,725 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,725 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:05,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:05,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:05,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:05,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14ccc3b6[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:05,729 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:05,729 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,730 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,731 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:05,731 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:05,731 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:05,731 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,731 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:05,736 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,736 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:05,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:05,736 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:05,737 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:05,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:05,739 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:05,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:05,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:05,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:05,739 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:05,741 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,741 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:05,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:05,742 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:05,742 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,744 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,747 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:05,747 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:05,747 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:05,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:05,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:05,747 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,747 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:05,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:05,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:05,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:05,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:05,747 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,747 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:05,747 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:05,848 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:05,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:54:05,849 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:54:05,850 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:05,850 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:05,850 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:05,850 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:05,850 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:05,850 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:05,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:05,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:05,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:05,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:05,855 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:05,855 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:05,855 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:05,855 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:05,856 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,856 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,857 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,858 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:05,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,858 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,858 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:05,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:05,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:05,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:05,866 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:05,866 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:05,866 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:05,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681095245861, max timestamp=1681095245875Num uniques 859; 2023-04-10 02:54:05,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681095245861, max timestamp=1681095245875Num uniques 859; 2023-04-10 02:54:05,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-10 02:54:05,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095245865, max timestamp=1681095245876Num uniques 859; 2023-04-10 02:54:05,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:54:05,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-10 02:54:05,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:05,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:05,882 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:05,882 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:05,882 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:05,882 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:05,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:05,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:05,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:05,883 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:05,883 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/3a25e47bd65c4d6ba8f493e5485cf4ab 2023-04-10 02:54:05,885 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,885 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:05,885 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,885 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:05,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:05,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1887a800[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:05,889 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:05,889 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,891 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,891 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:05,891 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:05,891 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:05,892 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:05,892 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:05,892 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:05,893 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,893 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:05,893 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:05,893 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:05,893 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:05,893 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:05,893 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:05,893 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:05,899 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,899 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:05,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:05,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,899 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:05,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:05,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:05,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:05,900 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:05,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,902 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:05,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:05,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:05,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095305897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095305897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,906 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:05,906 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:05,906 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:05,906 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,906 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:05,906 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:05,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:05,906 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:05,906 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,907 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:05,907 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:05,907 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:05,907 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:05,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:05,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:05,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095305905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:05,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:05,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095305905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,007 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:06,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:06,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:06,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095306006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:06,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095306006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:06,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095306010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:06,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095306011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:06,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:06,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:06,022 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:06,022 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,022 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:06,022 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:06,024 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,024 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,026 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:06,026 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:06,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:06,027 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,027 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:06,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:06,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:06,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:06,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:06,029 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:06,029 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,029 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:06,037 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:06,037 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:06,037 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,037 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:06,038 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:06,038 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,038 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:06,038 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:06,038 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,038 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:06,038 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:06,039 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,040 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,040 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:06,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:06,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70aae97f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:06,042 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:06,042 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,043 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,044 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:06,044 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:06,044 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:06,045 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,045 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:06,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:06,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:06,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:06,047 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:06,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:06,050 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:06,050 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:06,050 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:06,050 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:06,050 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:06,052 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,052 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,052 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:06,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:06,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:06,053 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,054 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,056 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,056 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,056 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,056 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,056 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:06,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:06,056 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:06,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:06,056 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:06,057 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:06,056 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,057 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,057 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:06,057 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:06,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:06,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:06,157 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:06,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:06,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:06,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:06,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:06,161 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:06,161 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,161 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:06,161 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:06,162 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,162 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,163 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:06,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:06,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:06,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:06,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:06,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:06,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:06,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:06,165 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:06,165 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,165 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:06,165 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:06,165 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:06,165 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,165 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:06,166 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:06,166 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,166 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:06,166 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:06,166 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,166 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:06,166 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:06,169 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,169 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,169 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,169 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:06,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@784fda82[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:06,171 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:06,171 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,172 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,172 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:06,172 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:06,172 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,172 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:06,172 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:06,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,173 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:06,173 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:06,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,173 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:06,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:06,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:06,176 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:06,176 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:06,176 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:06,176 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:06,176 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:06,178 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,178 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:06,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:06,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:06,178 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,180 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,181 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,192 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:06,193 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:06,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:06,193 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:06,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:06,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:06,193 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:06,193 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,193 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,193 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,193 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,193 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,193 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,193 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:06,193 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:06,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:06,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:06,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095306213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:06,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095306217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:06,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095306217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:06,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095306224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,293 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:06,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:06,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:06,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:06,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:06,297 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:06,297 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,297 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:06,297 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:06,298 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,298 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:06,299 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:06,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:06,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,300 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,300 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:06,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:06,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:06,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:06,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:06,301 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:06,301 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,301 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:06,301 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:06,301 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:06,302 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,302 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:06,302 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:06,302 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,302 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:06,302 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:06,302 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,302 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:06,302 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:06,303 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,303 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,304 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,304 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:06,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:06,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aec774e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:06,306 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:06,306 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,307 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:06,308 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:06,308 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,308 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:06,311 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/6b4f36250b2f4619aaf9c4575c3b43d0 2023-04-10 02:54:06,311 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:06,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:06,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:06,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:06,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:06,314 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:06,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:06,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:06,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:06,314 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:06,320 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,320 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:06,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,320 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:06,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,322 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,325 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,333 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,333 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,333 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,333 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:06,333 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:06,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:06,333 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:06,333 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:06,333 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:06,333 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:06,333 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,333 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,333 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:06,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:06,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:06,433 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:06,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:06,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:06,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:06,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:06,437 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:06,437 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,437 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:06,437 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:06,437 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,437 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,439 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:06,439 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,439 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,439 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:06,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:06,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:06,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:06,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:06,440 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:06,441 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,441 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:06,441 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:06,441 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:06,444 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,444 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,444 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,444 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:06,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f8e19bb[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:06,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:06,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,451 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:06,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:06,451 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,451 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:06,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:06,453 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:06,453 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:06,453 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,453 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:06,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:06,455 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:06,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:06,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:06,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:06,455 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:06,457 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,457 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:06,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,457 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,458 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,463 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:06,463 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,463 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:06,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:06,463 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,463 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:06,463 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:06,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:06,463 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,464 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:06,464 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:06,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:06,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:06,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095306517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:06,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095306525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:06,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095306525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:06,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095306528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,564 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:06,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:06,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:06,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:06,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:06,567 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:06,567 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,568 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:06,568 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:06,568 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,568 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:06,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:06,570 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,570 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:06,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:06,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:06,571 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:06,571 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,571 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:06,572 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:06,572 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:06,572 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,572 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:06,572 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:06,572 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,573 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:06,573 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:06,573 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,573 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:06,573 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:06,574 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:06,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@848fc24[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:06,577 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:06,577 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,578 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,578 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:06,578 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,578 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:06,578 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:06,578 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:06,581 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,581 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:06,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,581 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:06,581 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:06,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:06,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:06,587 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:06,588 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:06,588 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:06,588 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:06,588 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:06,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:06,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,591 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,592 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,597 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,597 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:06,597 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:06,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:06,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:06,597 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,597 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:06,597 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:06,597 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,597 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:06,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:06,597 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:06,597 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,598 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,598 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:06,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:06,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:06,698 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:06,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:06,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:06,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:06,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:06,704 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:06,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:06,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:06,705 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,705 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,711 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,711 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:06,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:06,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:06,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,711 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,712 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:06,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:06,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:06,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:06,713 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:06,713 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,713 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:06,714 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:06,714 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:06,714 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,714 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:06,714 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:06,714 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,714 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:06,714 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:06,714 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,714 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:06,714 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:06,716 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,716 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:06,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed5bc65[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:06,719 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:06,719 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,720 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,720 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:06,721 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:06,721 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:06,721 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,721 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:06,722 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,722 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:06,722 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,722 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:06,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:06,728 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:06,728 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:06,728 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:06,729 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:06,729 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:06,732 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,732 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:06,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,732 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:06,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:06,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:06,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:06,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:06,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:06,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,735 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:06,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:06,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:06,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:06,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:06,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:06,739 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:06,739 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:06,738 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:06,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:06,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:06,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:06,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:06,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:06,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/899e69458d1e4f78b3cc4cf0fb11b31c 2023-04-10 02:54:06,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/3a25e47bd65c4d6ba8f493e5485cf4ab as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3a25e47bd65c4d6ba8f493e5485cf4ab 2023-04-10 02:54:06,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3a25e47bd65c4d6ba8f493e5485cf4ab, entries=150, sequenceid=640, filesize=11.9 K 2023-04-10 02:54:06,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/6b4f36250b2f4619aaf9c4575c3b43d0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6b4f36250b2f4619aaf9c4575c3b43d0 2023-04-10 02:54:06,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6b4f36250b2f4619aaf9c4575c3b43d0, entries=150, sequenceid=640, filesize=11.9 K 2023-04-10 02:54:06,781 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/899e69458d1e4f78b3cc4cf0fb11b31c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/899e69458d1e4f78b3cc4cf0fb11b31c 2023-04-10 02:54:06,786 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/899e69458d1e4f78b3cc4cf0fb11b31c, entries=150, sequenceid=640, filesize=11.9 K 2023-04-10 02:54:06,787 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=136.42 KB/139690 for 66e59774c088f69ab4d8d1711037d595 in 938ms, sequenceid=640, compaction requested=true 2023-04-10 02:54:06,787 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:06,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:06,787 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:54:06,787 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:54:06,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:06,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:06,789 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47322 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:54:06,789 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/B is initiating minor compaction (all files) 2023-04-10 02:54:06,789 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/B in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,789 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/39cf26f3fae847bb9383854ad668f7ec, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/72fbf10769b6420d8ef667b7b9b0f5fd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/1866d6ee9a0642139b0c74609c95095b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6b4f36250b2f4619aaf9c4575c3b43d0] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=46.2 K 2023-04-10 02:54:06,789 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47322 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:54:06,789 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/A is initiating minor compaction (all files) 2023-04-10 02:54:06,789 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/A in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,790 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3ac6b7b7da7848a299866825d94eea48, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/614fa69df5f143a394b683b97bd4849a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/07725401a83b43b3b25199208f7c31cb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3a25e47bd65c4d6ba8f493e5485cf4ab] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=46.2 K 2023-04-10 02:54:06,790 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 39cf26f3fae847bb9383854ad668f7ec, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681095243040 2023-04-10 02:54:06,790 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 3ac6b7b7da7848a299866825d94eea48, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681095243040 2023-04-10 02:54:06,791 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 72fbf10769b6420d8ef667b7b9b0f5fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1681095243096 2023-04-10 02:54:06,791 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 614fa69df5f143a394b683b97bd4849a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1681095243096 2023-04-10 02:54:06,791 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 1866d6ee9a0642139b0c74609c95095b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1681095243716 2023-04-10 02:54:06,791 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 07725401a83b43b3b25199208f7c31cb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1681095243716 2023-04-10 02:54:06,792 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6b4f36250b2f4619aaf9c4575c3b43d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681095245847 2023-04-10 02:54:06,792 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 3a25e47bd65c4d6ba8f493e5485cf4ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681095245847 2023-04-10 02:54:06,804 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#A#compaction#102 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:06,821 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#B#compaction#103 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:06,839 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:06,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:06,842 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/35e69a84733642a2bfb3ed4994f4be98 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/35e69a84733642a2bfb3ed4994f4be98 2023-04-10 02:54:06,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:06,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:06,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:06,846 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:06,846 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,846 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:06,846 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:06,848 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,848 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,849 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:06,849 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:06,850 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:06,850 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:06,850 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/B of 66e59774c088f69ab4d8d1711037d595 into 35e69a84733642a2bfb3ed4994f4be98(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:06,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:06,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:06,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:06,852 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:06,851 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:06,852 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:06,852 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/B, priority=12, startTime=1681095246787; duration=0sec 2023-04-10 02:54:06,852 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:06,852 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:06,852 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:54:06,854 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47322 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:54:06,854 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/C is initiating minor compaction (all files) 2023-04-10 02:54:06,854 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/C in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,854 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/844c503a66ff431db13ed21e657c7e5f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/de40f81f673b465d92b5d8cfae4494f5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/d8266c5a26c0492ba602f3f00583c044, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/899e69458d1e4f78b3cc4cf0fb11b31c] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=46.2 K 2023-04-10 02:54:06,855 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 844c503a66ff431db13ed21e657c7e5f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681095243040 2023-04-10 02:54:06,855 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting de40f81f673b465d92b5d8cfae4494f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1681095243096 2023-04-10 02:54:06,856 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d8266c5a26c0492ba602f3f00583c044, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1681095243716 2023-04-10 02:54:06,856 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 899e69458d1e4f78b3cc4cf0fb11b31c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681095245847 2023-04-10 02:54:06,860 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:06,860 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:06,860 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:06,860 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:06,860 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=136.42 KB heapSize=376 KB 2023-04-10 02:54:06,860 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:06,860 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:06,861 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:06,861 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:06,861 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:06,861 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:06,869 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#C#compaction#105 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:06,876 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/cb7f1cab724f4f8ea825ef451405f53a 2023-04-10 02:54:06,900 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/37f423378e2949ddb3488477c301ea39 2023-04-10 02:54:06,911 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/6adb1f8e517d4cd6a65d5b46344eecff as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/6adb1f8e517d4cd6a65d5b46344eecff 2023-04-10 02:54:06,922 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/C of 66e59774c088f69ab4d8d1711037d595 into 6adb1f8e517d4cd6a65d5b46344eecff(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:06,922 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:06,922 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/C, priority=12, startTime=1681095246787; duration=0sec 2023-04-10 02:54:06,922 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:06,980 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/687e663948204bcd8abe8e326ae464d2 2023-04-10 02:54:06,988 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/cb7f1cab724f4f8ea825ef451405f53a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/cb7f1cab724f4f8ea825ef451405f53a 2023-04-10 02:54:06,996 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/cb7f1cab724f4f8ea825ef451405f53a, entries=150, sequenceid=675, filesize=11.9 K 2023-04-10 02:54:06,997 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/37f423378e2949ddb3488477c301ea39 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/37f423378e2949ddb3488477c301ea39 2023-04-10 02:54:07,001 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/37f423378e2949ddb3488477c301ea39, entries=150, sequenceid=675, filesize=11.9 K 2023-04-10 02:54:07,003 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/687e663948204bcd8abe8e326ae464d2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/687e663948204bcd8abe8e326ae464d2 2023-04-10 02:54:07,008 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/687e663948204bcd8abe8e326ae464d2, entries=150, sequenceid=675, filesize=11.9 K 2023-04-10 02:54:07,009 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~375.95 KB/384976, currentSize=0 B/0 for 66e59774c088f69ab4d8d1711037d595 in 149ms, sequenceid=675, compaction requested=false 2023-04-10 02:54:07,009 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:07,009 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:07,009 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:07,009 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:07,009 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:07,013 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,014 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:07,014 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,014 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:07,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:07,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67eafcf6[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:07,016 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:07,016 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,023 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,023 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:07,023 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:07,023 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:07,023 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,023 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:07,025 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:07,025 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,025 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:07,025 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:07,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:07,028 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:07,028 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:07,029 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:07,029 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:07,029 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:07,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:07,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:07,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:07,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,032 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:07,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:54:07,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:54:07,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,034 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:07,034 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:07,034 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:07,034 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:07,035 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:07,035 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:07,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:07,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,036 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:07,036 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:07,037 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:07,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:07,037 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,037 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,037 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,037 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,037 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,037 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:07,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:07,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:07,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:07,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:07,037 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:07,037 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:07,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:07,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681095247046, max timestamp=1681095247065Num uniques 859; 2023-04-10 02:54:07,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:54:07,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681095247046, max timestamp=1681095247065Num uniques 859; 2023-04-10 02:54:07,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:54:07,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681095247046, max timestamp=1681095247065Num uniques 859; 2023-04-10 02:54:07,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:54:07,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095307096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095307114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095307114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095307114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,138 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:07,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:07,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:07,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:07,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:07,142 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:07,142 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:07,142 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:07,142 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:07,142 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,142 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:07,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:07,145 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,146 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,146 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:07,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:07,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:07,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:07,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:07,146 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:07,146 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:07,147 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:07,147 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:07,147 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:07,147 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:07,147 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:07,147 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:07,147 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:07,147 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:07,147 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:07,147 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,147 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:07,147 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:07,149 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,149 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:07,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:07,150 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,150 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:07,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:07,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:07,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:07,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:07,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38766271[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:07,151 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:07,151 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,152 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,152 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:07,152 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:07,152 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:07,153 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,153 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:07,154 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:07,154 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:07,154 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:07,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:07,154 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:07,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:07,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:07,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:07,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:07,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:07,158 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:07,158 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:07,158 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:07,158 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:07,158 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:07,161 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,161 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:07,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:07,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,161 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:07,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:07,161 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:07,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:07,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:07,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,162 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:07,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:07,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:07,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:07,165 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:07,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:07,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:07,165 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:07,165 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:07,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:07,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:07,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:07,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:07,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095307218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095307223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095307223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095307223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,229 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/b80409d4b21b41b89e82b9f803382d0a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/b80409d4b21b41b89e82b9f803382d0a 2023-04-10 02:54:07,235 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/A of 66e59774c088f69ab4d8d1711037d595 into b80409d4b21b41b89e82b9f803382d0a(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:07,235 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:07,236 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/A, priority=12, startTime=1681095246787; duration=0sec 2023-04-10 02:54:07,236 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:07,266 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:07,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:07,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:07,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:07,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:07,282 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:07,282 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:07,282 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:07,282 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:07,282 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,282 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,286 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:07,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:07,286 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,286 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,286 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:07,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:07,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:07,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:07,288 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:07,288 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:07,288 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:07,292 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:07,292 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:07,292 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:07,292 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:07,292 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:07,292 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:07,292 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:07,293 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:07,293 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,293 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:07,293 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:07,295 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:07,295 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:07,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:07,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:07,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:07,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77101087[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:07,299 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:07,299 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,300 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,300 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:07,300 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:07,300 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:07,300 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,301 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:07,302 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,302 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:07,302 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:07,302 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:07,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:07,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:07,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:07,305 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:07,305 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:07,305 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:07,305 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:07,305 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:07,306 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,306 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:07,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:07,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:07,307 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,308 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:07,309 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:07,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,317 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,317 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:07,317 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:07,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:07,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:07,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:07,317 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:07,317 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,318 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,318 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,318 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,318 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,318 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:07,318 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:07,317 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:07,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:07,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:07,418 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:07,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:07,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:07,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095307422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:07,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:07,424 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:07,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:07,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:07,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:07,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095307425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095307425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,428 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,428 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095307426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,429 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,429 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:07,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:07,430 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,430 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:07,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:07,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:07,431 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:07,431 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:07,431 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:07,431 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:07,431 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:07,431 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:07,431 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:07,431 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:07,431 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:07,431 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:07,432 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:07,432 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,432 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:07,432 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:07,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,433 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:07,433 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,434 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:07,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:07,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5405ac94[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:07,435 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:07,435 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,436 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,436 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:07,436 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:07,436 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:07,437 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,437 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:07,439 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:07,440 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:07,440 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:07,440 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:07,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:07,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:07,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:07,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/05f6ac724b364fcd821f26906c1609c6 2023-04-10 02:54:07,445 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:07,445 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:07,445 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:07,445 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:07,445 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:07,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:07,452 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,452 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:07,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,452 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:07,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:07,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:07,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:07,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:07,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:07,458 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:07,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:07,461 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,463 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:07,463 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,463 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:07,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:07,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:07,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:07,463 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:07,463 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:07,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:07,464 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:07,464 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:07,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:07,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:07,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/88a92fe94d40448eb6dbd8536b4a2229 2023-04-10 02:54:07,481 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/e11b7b67629f4dc4a8069696f2b4318a 2023-04-10 02:54:07,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/05f6ac724b364fcd821f26906c1609c6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/05f6ac724b364fcd821f26906c1609c6 2023-04-10 02:54:07,493 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/05f6ac724b364fcd821f26906c1609c6, entries=250, sequenceid=687, filesize=16.7 K 2023-04-10 02:54:07,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/88a92fe94d40448eb6dbd8536b4a2229 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/88a92fe94d40448eb6dbd8536b4a2229 2023-04-10 02:54:07,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/88a92fe94d40448eb6dbd8536b4a2229, entries=150, sequenceid=687, filesize=11.9 K 2023-04-10 02:54:07,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/e11b7b67629f4dc4a8069696f2b4318a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e11b7b67629f4dc4a8069696f2b4318a 2023-04-10 02:54:07,504 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e11b7b67629f4dc4a8069696f2b4318a, entries=150, sequenceid=687, filesize=11.9 K 2023-04-10 02:54:07,505 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 66e59774c088f69ab4d8d1711037d595 in 472ms, sequenceid=687, compaction requested=true 2023-04-10 02:54:07,505 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:07,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:07,506 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:07,506 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:07,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:07,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:07,507 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42596 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:07,507 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/A is initiating minor compaction (all files) 2023-04-10 02:54:07,507 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:07,507 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/A in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:07,507 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/B is initiating minor compaction (all files) 2023-04-10 02:54:07,507 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/b80409d4b21b41b89e82b9f803382d0a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/cb7f1cab724f4f8ea825ef451405f53a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/05f6ac724b364fcd821f26906c1609c6] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=41.6 K 2023-04-10 02:54:07,507 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/B in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:07,507 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/35e69a84733642a2bfb3ed4994f4be98, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/37f423378e2949ddb3488477c301ea39, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/88a92fe94d40448eb6dbd8536b4a2229] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=36.8 K 2023-04-10 02:54:07,507 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting b80409d4b21b41b89e82b9f803382d0a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681095245847 2023-04-10 02:54:07,507 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 35e69a84733642a2bfb3ed4994f4be98, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681095245847 2023-04-10 02:54:07,508 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting cb7f1cab724f4f8ea825ef451405f53a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1681095245893 2023-04-10 02:54:07,508 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 37f423378e2949ddb3488477c301ea39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1681095245893 2023-04-10 02:54:07,508 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 05f6ac724b364fcd821f26906c1609c6, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681095247027 2023-04-10 02:54:07,508 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 88a92fe94d40448eb6dbd8536b4a2229, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681095247031 2023-04-10 02:54:07,518 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#B#compaction#111 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:07,533 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#A#compaction#112 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:07,553 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/540b40d203fb4fc3a72ecd29caee64d5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/540b40d203fb4fc3a72ecd29caee64d5 2023-04-10 02:54:07,560 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/B of 66e59774c088f69ab4d8d1711037d595 into 540b40d203fb4fc3a72ecd29caee64d5(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:07,560 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:07,560 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/B, priority=13, startTime=1681095247506; duration=0sec 2023-04-10 02:54:07,560 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:07,560 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:07,561 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:07,561 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/C is initiating minor compaction (all files) 2023-04-10 02:54:07,562 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/C in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:07,562 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/6adb1f8e517d4cd6a65d5b46344eecff, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/687e663948204bcd8abe8e326ae464d2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e11b7b67629f4dc4a8069696f2b4318a] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=36.8 K 2023-04-10 02:54:07,562 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6adb1f8e517d4cd6a65d5b46344eecff, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681095245847 2023-04-10 02:54:07,562 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 687e663948204bcd8abe8e326ae464d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1681095245893 2023-04-10 02:54:07,563 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting e11b7b67629f4dc4a8069696f2b4318a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681095247031 2023-04-10 02:54:07,564 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:07,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:07,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:07,572 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#C#compaction#113 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:07,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:07,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:07,572 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:07,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:07,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:07,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:07,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,574 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:07,574 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:07,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:07,574 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,574 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:07,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:07,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:07,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:07,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:07,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:07,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:07,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:07,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:07,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:07,581 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:07,581 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:07,581 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:07,581 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. started... 2023-04-10 02:54:07,581 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-10 02:54:07,581 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:07,581 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:07,581 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:07,581 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:07,582 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:07,582 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:07,587 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/44804488eed346f3891c4bc4da1bff9f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/44804488eed346f3891c4bc4da1bff9f 2023-04-10 02:54:07,593 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/C of 66e59774c088f69ab4d8d1711037d595 into 44804488eed346f3891c4bc4da1bff9f(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:07,593 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:07,593 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/C, priority=13, startTime=1681095247506; duration=0sec 2023-04-10 02:54:07,594 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:07,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:54:07,728 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. as already flushing 2023-04-10 02:54:07,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095307734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095307734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095307735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095307735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,832 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6129209e to 127.0.0.1:60568 2023-04-10 02:54:07,832 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1818ad0a to 127.0.0.1:60568 2023-04-10 02:54:07,832 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:07,832 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:07,836 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04fb2b82 to 127.0.0.1:60568 2023-04-10 02:54:07,837 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:07,838 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4b02eeb3 to 127.0.0.1:60568 2023-04-10 02:54:07,838 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:07,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095307839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095307839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095307840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:07,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095307843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:07,966 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/7234bec04855405ca99426ab3cbfdbeb as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/7234bec04855405ca99426ab3cbfdbeb 2023-04-10 02:54:07,972 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/A of 66e59774c088f69ab4d8d1711037d595 into 7234bec04855405ca99426ab3cbfdbeb(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:07,972 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:07,972 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/A, priority=13, startTime=1681095247505; duration=0sec 2023-04-10 02:54:07,972 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:07,998 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/00f2cc9a1683476c853885816453d6d4 2023-04-10 02:54:08,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:08,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:08,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:08,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095308041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095308041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095308041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:08,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095308045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:08,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:08,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095308344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095308344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:08,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095308345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:08,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095308348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,425 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/0147b5751c174844b4371001d9028737 2023-04-10 02:54:08,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:08,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55592 deadline: 1681095308645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,647 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8186 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:54:08,840 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/32b0194bf8f24f519ab00008b0d16745 2023-04-10 02:54:08,844 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/00f2cc9a1683476c853885816453d6d4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/00f2cc9a1683476c853885816453d6d4 2023-04-10 02:54:08,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:08,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55636 deadline: 1681095308847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,848 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/00f2cc9a1683476c853885816453d6d4, entries=150, sequenceid=725, filesize=11.9 K 2023-04-10 02:54:08,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:08,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55614 deadline: 1681095308848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,849 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/0147b5751c174844b4371001d9028737 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/0147b5751c174844b4371001d9028737 2023-04-10 02:54:08,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:08,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55622 deadline: 1681095308849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:08,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55662 deadline: 1681095308853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,854 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/0147b5751c174844b4371001d9028737, entries=150, sequenceid=725, filesize=11.9 K 2023-04-10 02:54:08,855 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/32b0194bf8f24f519ab00008b0d16745 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/32b0194bf8f24f519ab00008b0d16745 2023-04-10 02:54:08,859 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/32b0194bf8f24f519ab00008b0d16745, entries=150, sequenceid=725, filesize=11.9 K 2023-04-10 02:54:08,859 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 66e59774c088f69ab4d8d1711037d595 in 1278ms, sequenceid=725, compaction requested=false 2023-04-10 02:54:08,859 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:08,859 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:08,861 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,861 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:08,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:08,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:08,862 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:08,862 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:08,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:08,863 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:08,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64b8f7ae[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:08,863 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:08,864 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:08,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:08,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:08,865 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:08,865 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:08,865 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:08,865 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,865 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:08,867 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:08,867 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,867 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:08,867 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:08,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:08,870 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:08,870 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:08,870 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:08,870 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:08,870 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:08,871 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:08,871 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:08,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:08,871 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:08,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:08,872 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:08,873 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:08,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:08,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:08,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:08,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:08,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:08,875 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:08,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:08,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:08,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:08,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:08,875 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:08,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:08,875 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:08,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:08,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:08,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:08,976 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:08,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:09,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:54:09,852 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:54:09,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:09,852 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e728705 to 127.0.0.1:60568 2023-04-10 02:54:09,853 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:09,853 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:09,853 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:09,853 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:09,853 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:09,853 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:09,854 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b886c8f to 127.0.0.1:60568 2023-04-10 02:54:09,854 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:09,858 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09a575b8 to 127.0.0.1:60568 2023-04-10 02:54:09,858 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:09,860 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77648431 to 127.0.0.1:60568 2023-04-10 02:54:09,860 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:10,261 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/3016959fc2214ba9a403a94c6d236371 2023-04-10 02:54:10,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/6c13c3e03b7f4729b02afa5d63ed0b92 2023-04-10 02:54:11,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/6f7626207d354561b88b757deab32d55 2023-04-10 02:54:11,092 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/3016959fc2214ba9a403a94c6d236371 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3016959fc2214ba9a403a94c6d236371 2023-04-10 02:54:11,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3016959fc2214ba9a403a94c6d236371, entries=150, sequenceid=738, filesize=11.9 K 2023-04-10 02:54:11,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/6c13c3e03b7f4729b02afa5d63ed0b92 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6c13c3e03b7f4729b02afa5d63ed0b92 2023-04-10 02:54:11,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6c13c3e03b7f4729b02afa5d63ed0b92, entries=150, sequenceid=738, filesize=11.9 K 2023-04-10 02:54:11,101 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/6f7626207d354561b88b757deab32d55 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/6f7626207d354561b88b757deab32d55 2023-04-10 02:54:11,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/6f7626207d354561b88b757deab32d55, entries=150, sequenceid=738, filesize=11.9 K 2023-04-10 02:54:11,106 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=20.13 KB/20610 for 66e59774c088f69ab4d8d1711037d595 in 1254ms, sequenceid=738, compaction requested=true 2023-04-10 02:54:11,106 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:11,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:11,106 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:11,106 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:11,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:11,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:11,107 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:11,107 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:11,107 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/A is initiating minor compaction (all files) 2023-04-10 02:54:11,107 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/B is initiating minor compaction (all files) 2023-04-10 02:54:11,107 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/A in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:11,107 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/B in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:11,108 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/7234bec04855405ca99426ab3cbfdbeb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/00f2cc9a1683476c853885816453d6d4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3016959fc2214ba9a403a94c6d236371] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=36.9 K 2023-04-10 02:54:11,108 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/540b40d203fb4fc3a72ecd29caee64d5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/0147b5751c174844b4371001d9028737, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6c13c3e03b7f4729b02afa5d63ed0b92] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=36.9 K 2023-04-10 02:54:11,108 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 7234bec04855405ca99426ab3cbfdbeb, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681095247031 2023-04-10 02:54:11,108 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 540b40d203fb4fc3a72ecd29caee64d5, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681095247031 2023-04-10 02:54:11,108 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 00f2cc9a1683476c853885816453d6d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=725, earliestPutTs=1681095247085 2023-04-10 02:54:11,108 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 0147b5751c174844b4371001d9028737, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=725, earliestPutTs=1681095247085 2023-04-10 02:54:11,109 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 3016959fc2214ba9a403a94c6d236371, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=738, earliestPutTs=1681095247732 2023-04-10 02:54:11,109 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6c13c3e03b7f4729b02afa5d63ed0b92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=738, earliestPutTs=1681095247732 2023-04-10 02:54:11,117 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#B#compaction#120 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:11,119 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#A#compaction#121 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:11,529 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/2b7d5689f9da488e82d0b04b9aa1df1c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2b7d5689f9da488e82d0b04b9aa1df1c 2023-04-10 02:54:11,529 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/1097ca66a7eb4ebfbeeb2e26adf2ef77 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/1097ca66a7eb4ebfbeeb2e26adf2ef77 2023-04-10 02:54:11,534 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/B of 66e59774c088f69ab4d8d1711037d595 into 1097ca66a7eb4ebfbeeb2e26adf2ef77(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:11,534 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/A of 66e59774c088f69ab4d8d1711037d595 into 2b7d5689f9da488e82d0b04b9aa1df1c(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:11,534 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:11,534 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:11,534 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/B, priority=13, startTime=1681095251106; duration=0sec 2023-04-10 02:54:11,534 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/A, priority=13, startTime=1681095251106; duration=0sec 2023-04-10 02:54:11,534 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:11,534 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:11,534 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:11,535 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:11,535 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 66e59774c088f69ab4d8d1711037d595/C is initiating minor compaction (all files) 2023-04-10 02:54:11,535 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 66e59774c088f69ab4d8d1711037d595/C in TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:11,536 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/44804488eed346f3891c4bc4da1bff9f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/32b0194bf8f24f519ab00008b0d16745, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/6f7626207d354561b88b757deab32d55] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp, totalSize=36.9 K 2023-04-10 02:54:11,536 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 44804488eed346f3891c4bc4da1bff9f, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681095247031 2023-04-10 02:54:11,536 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 32b0194bf8f24f519ab00008b0d16745, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=725, earliestPutTs=1681095247085 2023-04-10 02:54:11,537 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6f7626207d354561b88b757deab32d55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=738, earliestPutTs=1681095247732 2023-04-10 02:54:11,543 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 66e59774c088f69ab4d8d1711037d595#C#compaction#122 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:11,952 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/e3f3c59df6434964a033aafcd72b8391 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e3f3c59df6434964a033aafcd72b8391 2023-04-10 02:54:11,957 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 66e59774c088f69ab4d8d1711037d595/C of 66e59774c088f69ab4d8d1711037d595 into e3f3c59df6434964a033aafcd72b8391(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:11,957 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:11,957 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595., storeName=66e59774c088f69ab4d8d1711037d595/C, priority=13, startTime=1681095251106; duration=0sec 2023-04-10 02:54:11,957 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:18,681 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e16d41b to 127.0.0.1:60568 2023-04-10 02:54:18,681 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:18,681 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-10 02:54:18,681 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 135 2023-04-10 02:54:18,681 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 87 2023-04-10 02:54:18,681 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 147 2023-04-10 02:54:18,681 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 130 2023-04-10 02:54:18,681 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 121 2023-04-10 02:54:18,681 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-10 02:54:18,681 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 4495 2023-04-10 02:54:18,682 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 4378 2023-04-10 02:54:18,682 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-10 02:54:18,682 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 1961 2023-04-10 02:54:18,682 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 5878 rows 2023-04-10 02:54:18,682 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 1955 2023-04-10 02:54:18,682 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 5865 rows 2023-04-10 02:54:18,682 INFO [Listener at localhost/44533] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-10 02:54:18,682 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2678d495 to 127.0.0.1:60568 2023-04-10 02:54:18,682 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:18,689 INFO [Listener at localhost/44533] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-10 02:54:18,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-10 02:54:18,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:18,705 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095258704"}]},"ts":"1681095258704"} 2023-04-10 02:54:18,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-10 02:54:18,706 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-10 02:54:18,709 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-10 02:54:18,710 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=66e59774c088f69ab4d8d1711037d595, UNASSIGN}] 2023-04-10 02:54:18,711 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=66e59774c088f69ab4d8d1711037d595, UNASSIGN 2023-04-10 02:54:18,712 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=66e59774c088f69ab4d8d1711037d595, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:18,712 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095258712"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095258712"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095258712"}]},"ts":"1681095258712"} 2023-04-10 02:54:18,714 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:54:18,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-10 02:54:18,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:54:18,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 66e59774c088f69ab4d8d1711037d595, disabling compactions & flushes 2023-04-10 02:54:18,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:18,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:18,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. after waiting 0 ms 2023-04-10 02:54:18,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:18,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 66e59774c088f69ab4d8d1711037d595 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-10 02:54:18,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=A 2023-04-10 02:54:18,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:18,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=B 2023-04-10 02:54:18,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:18,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 66e59774c088f69ab4d8d1711037d595, store=C 2023-04-10 02:54:18,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:19,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-10 02:54:19,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/5c921888158049a88169647ab13aa203 2023-04-10 02:54:19,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-10 02:54:19,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/4ea93e94bf6847dc9f5901088ec7bea8 2023-04-10 02:54:19,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/3cf551abfa2e4603a95c01935fd011c0 2023-04-10 02:54:19,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/A/5c921888158049a88169647ab13aa203 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/5c921888158049a88169647ab13aa203 2023-04-10 02:54:19,722 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/5c921888158049a88169647ab13aa203, entries=100, sequenceid=748, filesize=9.5 K 2023-04-10 02:54:19,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/B/4ea93e94bf6847dc9f5901088ec7bea8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/4ea93e94bf6847dc9f5901088ec7bea8 2023-04-10 02:54:19,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/4ea93e94bf6847dc9f5901088ec7bea8, entries=100, sequenceid=748, filesize=9.5 K 2023-04-10 02:54:19,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/.tmp/C/3cf551abfa2e4603a95c01935fd011c0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/3cf551abfa2e4603a95c01935fd011c0 2023-04-10 02:54:19,735 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/3cf551abfa2e4603a95c01935fd011c0, entries=100, sequenceid=748, filesize=9.5 K 2023-04-10 02:54:19,736 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 66e59774c088f69ab4d8d1711037d595 in 863ms, sequenceid=748, compaction requested=false 2023-04-10 02:54:19,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8e8af2f0ccd54f9cb97bfe0a86fc0879, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/234552b740184d949b73be731d42f4f6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/c6410cd4221448ba995ca15cc97d6b82, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/29abaa7cf546448ca92cb1f2660f7393, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/b868fe15411c4447b4ff1449750d1721, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/ef21551b225e43b0acbac4e9fb521bc8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/ccdb3aeb7aca43b8995fa829ec56c799, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/573808cadbd449b29a960ca24604a7bf, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/eb5b2fd9463e460fa87c71c863283b0e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8b1c360ca9d14090bfef37d53ce8a4d6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/51b56cf3ba4841a391099b6e27bc7661, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/869c206eaba946c1a6754ea065e2862a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/dd8edfc2f00d4582b8c537e2826d94b8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8be63199820144f4ab4839bfdf1b7b4d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2abc1bfb5b3b4032a858b2473c549e62, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/f6340540963648bf85a81d2ffa867cd2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a0c5aaafde274aac88811b17006e30d2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/66e90230189e4c6396622355fb7165ff, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/7215e69ca87244fb92668c9c774de9a5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2ec25b07a2f34a14b0e8943ff59cbde2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a78e14b744c4418b954c702dcb55eba6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a8c95e587fd44aa4a5f4663ab0039355, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/29b3160e21bc491d8c6186cca8dc8a55, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2d7e36f6d7c64f06bc5e248a59cb75dc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/525dc1c10b1c453da2c19643fca20116, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/309ea6d8d8cb41f6b63e96986ebd1721, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/00feab6c288d4ff59efe72c924c8c084, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/92b42e0b61e94489ba411c9dcbc38c11, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/6fb4240684c248f1b7b4328aca8f70c5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3ac6b7b7da7848a299866825d94eea48, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/e3427564eb7b4048aa99475c9ad4b8dd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/614fa69df5f143a394b683b97bd4849a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/07725401a83b43b3b25199208f7c31cb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/b80409d4b21b41b89e82b9f803382d0a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3a25e47bd65c4d6ba8f493e5485cf4ab, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/cb7f1cab724f4f8ea825ef451405f53a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/05f6ac724b364fcd821f26906c1609c6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/7234bec04855405ca99426ab3cbfdbeb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/00f2cc9a1683476c853885816453d6d4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3016959fc2214ba9a403a94c6d236371] to archive 2023-04-10 02:54:19,761 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:54:19,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8e8af2f0ccd54f9cb97bfe0a86fc0879 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8e8af2f0ccd54f9cb97bfe0a86fc0879 2023-04-10 02:54:19,769 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/234552b740184d949b73be731d42f4f6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/234552b740184d949b73be731d42f4f6 2023-04-10 02:54:19,771 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/c6410cd4221448ba995ca15cc97d6b82 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/c6410cd4221448ba995ca15cc97d6b82 2023-04-10 02:54:19,772 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/29abaa7cf546448ca92cb1f2660f7393 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/29abaa7cf546448ca92cb1f2660f7393 2023-04-10 02:54:19,774 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/b868fe15411c4447b4ff1449750d1721 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/b868fe15411c4447b4ff1449750d1721 2023-04-10 02:54:19,776 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/ef21551b225e43b0acbac4e9fb521bc8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/ef21551b225e43b0acbac4e9fb521bc8 2023-04-10 02:54:19,777 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/ccdb3aeb7aca43b8995fa829ec56c799 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/ccdb3aeb7aca43b8995fa829ec56c799 2023-04-10 02:54:19,779 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/573808cadbd449b29a960ca24604a7bf to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/573808cadbd449b29a960ca24604a7bf 2023-04-10 02:54:19,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/eb5b2fd9463e460fa87c71c863283b0e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/eb5b2fd9463e460fa87c71c863283b0e 2023-04-10 02:54:19,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8b1c360ca9d14090bfef37d53ce8a4d6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8b1c360ca9d14090bfef37d53ce8a4d6 2023-04-10 02:54:19,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/51b56cf3ba4841a391099b6e27bc7661 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/51b56cf3ba4841a391099b6e27bc7661 2023-04-10 02:54:19,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/869c206eaba946c1a6754ea065e2862a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/869c206eaba946c1a6754ea065e2862a 2023-04-10 02:54:19,787 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/dd8edfc2f00d4582b8c537e2826d94b8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/dd8edfc2f00d4582b8c537e2826d94b8 2023-04-10 02:54:19,789 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8be63199820144f4ab4839bfdf1b7b4d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/8be63199820144f4ab4839bfdf1b7b4d 2023-04-10 02:54:19,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2abc1bfb5b3b4032a858b2473c549e62 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2abc1bfb5b3b4032a858b2473c549e62 2023-04-10 02:54:19,796 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/f6340540963648bf85a81d2ffa867cd2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/f6340540963648bf85a81d2ffa867cd2 2023-04-10 02:54:19,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a0c5aaafde274aac88811b17006e30d2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a0c5aaafde274aac88811b17006e30d2 2023-04-10 02:54:19,801 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/66e90230189e4c6396622355fb7165ff to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/66e90230189e4c6396622355fb7165ff 2023-04-10 02:54:19,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/7215e69ca87244fb92668c9c774de9a5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/7215e69ca87244fb92668c9c774de9a5 2023-04-10 02:54:19,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2ec25b07a2f34a14b0e8943ff59cbde2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2ec25b07a2f34a14b0e8943ff59cbde2 2023-04-10 02:54:19,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a78e14b744c4418b954c702dcb55eba6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a78e14b744c4418b954c702dcb55eba6 2023-04-10 02:54:19,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-10 02:54:19,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a8c95e587fd44aa4a5f4663ab0039355 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/a8c95e587fd44aa4a5f4663ab0039355 2023-04-10 02:54:19,811 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/29b3160e21bc491d8c6186cca8dc8a55 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/29b3160e21bc491d8c6186cca8dc8a55 2023-04-10 02:54:19,813 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2d7e36f6d7c64f06bc5e248a59cb75dc to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2d7e36f6d7c64f06bc5e248a59cb75dc 2023-04-10 02:54:19,815 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/525dc1c10b1c453da2c19643fca20116 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/525dc1c10b1c453da2c19643fca20116 2023-04-10 02:54:19,816 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/309ea6d8d8cb41f6b63e96986ebd1721 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/309ea6d8d8cb41f6b63e96986ebd1721 2023-04-10 02:54:19,817 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/00feab6c288d4ff59efe72c924c8c084 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/00feab6c288d4ff59efe72c924c8c084 2023-04-10 02:54:19,819 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/92b42e0b61e94489ba411c9dcbc38c11 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/92b42e0b61e94489ba411c9dcbc38c11 2023-04-10 02:54:19,820 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/6fb4240684c248f1b7b4328aca8f70c5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/6fb4240684c248f1b7b4328aca8f70c5 2023-04-10 02:54:19,822 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3ac6b7b7da7848a299866825d94eea48 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3ac6b7b7da7848a299866825d94eea48 2023-04-10 02:54:19,823 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/e3427564eb7b4048aa99475c9ad4b8dd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/e3427564eb7b4048aa99475c9ad4b8dd 2023-04-10 02:54:19,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/614fa69df5f143a394b683b97bd4849a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/614fa69df5f143a394b683b97bd4849a 2023-04-10 02:54:19,826 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/07725401a83b43b3b25199208f7c31cb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/07725401a83b43b3b25199208f7c31cb 2023-04-10 02:54:19,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/b80409d4b21b41b89e82b9f803382d0a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/b80409d4b21b41b89e82b9f803382d0a 2023-04-10 02:54:19,829 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3a25e47bd65c4d6ba8f493e5485cf4ab to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3a25e47bd65c4d6ba8f493e5485cf4ab 2023-04-10 02:54:19,830 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/cb7f1cab724f4f8ea825ef451405f53a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/cb7f1cab724f4f8ea825ef451405f53a 2023-04-10 02:54:19,831 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/05f6ac724b364fcd821f26906c1609c6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/05f6ac724b364fcd821f26906c1609c6 2023-04-10 02:54:19,833 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/7234bec04855405ca99426ab3cbfdbeb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/7234bec04855405ca99426ab3cbfdbeb 2023-04-10 02:54:19,834 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/00f2cc9a1683476c853885816453d6d4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/00f2cc9a1683476c853885816453d6d4 2023-04-10 02:54:19,836 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3016959fc2214ba9a403a94c6d236371 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/3016959fc2214ba9a403a94c6d236371 2023-04-10 02:54:19,862 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/40a738e6d1cd42f0a3dcc80d75af610f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/f5e60bb30882429b80b864ace4d21991, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/e900bbf601124deb9745d61e8be82617, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/cb5979e2d96346f787740e5e249a7c3e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/d7b3338fba0d4de7bb88a710d0015f9c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/24738c31b4464a5b80765965acb9ac75, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/91592184a3ab417fbd74f90a39c6e64f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/8acb20a965ff461f97b03af3b8e8ed8a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/43b3200facea4d33a5e52e34ac82c396, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/de5454143f5049fd8b7d54caa211257c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/62e9359c85df42e49ab0a740328acc3f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/fadbb6c8127c4bb29eefacdde2940763, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/31e316c6a9e24496a3dc824e462eb313, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/3acc6353436249b29f19910f4f0b3a4f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/fe031d88692b47deb17111dd2a1c66b6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/528dadaef65643638bc2949006c8afc3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6016299517cd4523940a8172fbaedcfb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/ab763019e6444286a6666bbc5be47bc2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/9d4d2bd708e742f79431023b97887e6d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/df48455438734f068f43216d91611f8e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/3301f68c06af4ad8895f4fe177e429f7, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/5c5d812ed84d4b0a90f8eb8da48c3ec8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7ff95fa517884da8a5af14f807d02d36, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/560bc97bb6914e2db48edf66824ba8b8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7e385518113443038b165a11da9fcd2c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/c75a3d13a2ad4403a17db41ce71afa2b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7837ee0803014440bf0dfa6c9ae988fd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/af7710fab0c046e5b2e4128608f1bce9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/5c844cd309c24bfcaed4a05a4562e84f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/39cf26f3fae847bb9383854ad668f7ec, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/78b1495d46c74c509de912e43fc9aaa5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/72fbf10769b6420d8ef667b7b9b0f5fd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/1866d6ee9a0642139b0c74609c95095b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/35e69a84733642a2bfb3ed4994f4be98, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6b4f36250b2f4619aaf9c4575c3b43d0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/37f423378e2949ddb3488477c301ea39, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/540b40d203fb4fc3a72ecd29caee64d5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/88a92fe94d40448eb6dbd8536b4a2229, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/0147b5751c174844b4371001d9028737, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6c13c3e03b7f4729b02afa5d63ed0b92] to archive 2023-04-10 02:54:19,863 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:54:19,864 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/40a738e6d1cd42f0a3dcc80d75af610f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/40a738e6d1cd42f0a3dcc80d75af610f 2023-04-10 02:54:19,866 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/f5e60bb30882429b80b864ace4d21991 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/f5e60bb30882429b80b864ace4d21991 2023-04-10 02:54:19,867 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/e900bbf601124deb9745d61e8be82617 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/e900bbf601124deb9745d61e8be82617 2023-04-10 02:54:19,868 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/cb5979e2d96346f787740e5e249a7c3e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/cb5979e2d96346f787740e5e249a7c3e 2023-04-10 02:54:19,869 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/d7b3338fba0d4de7bb88a710d0015f9c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/d7b3338fba0d4de7bb88a710d0015f9c 2023-04-10 02:54:19,870 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/24738c31b4464a5b80765965acb9ac75 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/24738c31b4464a5b80765965acb9ac75 2023-04-10 02:54:19,871 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/91592184a3ab417fbd74f90a39c6e64f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/91592184a3ab417fbd74f90a39c6e64f 2023-04-10 02:54:19,872 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/8acb20a965ff461f97b03af3b8e8ed8a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/8acb20a965ff461f97b03af3b8e8ed8a 2023-04-10 02:54:19,873 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/43b3200facea4d33a5e52e34ac82c396 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/43b3200facea4d33a5e52e34ac82c396 2023-04-10 02:54:19,874 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/de5454143f5049fd8b7d54caa211257c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/de5454143f5049fd8b7d54caa211257c 2023-04-10 02:54:19,875 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/62e9359c85df42e49ab0a740328acc3f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/62e9359c85df42e49ab0a740328acc3f 2023-04-10 02:54:19,876 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/fadbb6c8127c4bb29eefacdde2940763 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/fadbb6c8127c4bb29eefacdde2940763 2023-04-10 02:54:19,877 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/31e316c6a9e24496a3dc824e462eb313 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/31e316c6a9e24496a3dc824e462eb313 2023-04-10 02:54:19,879 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/3acc6353436249b29f19910f4f0b3a4f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/3acc6353436249b29f19910f4f0b3a4f 2023-04-10 02:54:19,880 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/fe031d88692b47deb17111dd2a1c66b6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/fe031d88692b47deb17111dd2a1c66b6 2023-04-10 02:54:19,881 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/528dadaef65643638bc2949006c8afc3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/528dadaef65643638bc2949006c8afc3 2023-04-10 02:54:19,882 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6016299517cd4523940a8172fbaedcfb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6016299517cd4523940a8172fbaedcfb 2023-04-10 02:54:19,883 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/ab763019e6444286a6666bbc5be47bc2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/ab763019e6444286a6666bbc5be47bc2 2023-04-10 02:54:19,884 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/9d4d2bd708e742f79431023b97887e6d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/9d4d2bd708e742f79431023b97887e6d 2023-04-10 02:54:19,885 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/df48455438734f068f43216d91611f8e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/df48455438734f068f43216d91611f8e 2023-04-10 02:54:19,886 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/3301f68c06af4ad8895f4fe177e429f7 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/3301f68c06af4ad8895f4fe177e429f7 2023-04-10 02:54:19,887 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/5c5d812ed84d4b0a90f8eb8da48c3ec8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/5c5d812ed84d4b0a90f8eb8da48c3ec8 2023-04-10 02:54:19,888 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7ff95fa517884da8a5af14f807d02d36 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7ff95fa517884da8a5af14f807d02d36 2023-04-10 02:54:19,889 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/560bc97bb6914e2db48edf66824ba8b8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/560bc97bb6914e2db48edf66824ba8b8 2023-04-10 02:54:19,890 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7e385518113443038b165a11da9fcd2c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7e385518113443038b165a11da9fcd2c 2023-04-10 02:54:19,891 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/c75a3d13a2ad4403a17db41ce71afa2b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/c75a3d13a2ad4403a17db41ce71afa2b 2023-04-10 02:54:19,892 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7837ee0803014440bf0dfa6c9ae988fd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/7837ee0803014440bf0dfa6c9ae988fd 2023-04-10 02:54:19,894 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/af7710fab0c046e5b2e4128608f1bce9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/af7710fab0c046e5b2e4128608f1bce9 2023-04-10 02:54:19,895 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/5c844cd309c24bfcaed4a05a4562e84f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/5c844cd309c24bfcaed4a05a4562e84f 2023-04-10 02:54:19,896 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/39cf26f3fae847bb9383854ad668f7ec to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/39cf26f3fae847bb9383854ad668f7ec 2023-04-10 02:54:19,897 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/78b1495d46c74c509de912e43fc9aaa5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/78b1495d46c74c509de912e43fc9aaa5 2023-04-10 02:54:19,898 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/72fbf10769b6420d8ef667b7b9b0f5fd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/72fbf10769b6420d8ef667b7b9b0f5fd 2023-04-10 02:54:19,899 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/1866d6ee9a0642139b0c74609c95095b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/1866d6ee9a0642139b0c74609c95095b 2023-04-10 02:54:19,900 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/35e69a84733642a2bfb3ed4994f4be98 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/35e69a84733642a2bfb3ed4994f4be98 2023-04-10 02:54:19,901 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6b4f36250b2f4619aaf9c4575c3b43d0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6b4f36250b2f4619aaf9c4575c3b43d0 2023-04-10 02:54:19,902 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/37f423378e2949ddb3488477c301ea39 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/37f423378e2949ddb3488477c301ea39 2023-04-10 02:54:19,903 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/540b40d203fb4fc3a72ecd29caee64d5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/540b40d203fb4fc3a72ecd29caee64d5 2023-04-10 02:54:19,905 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/88a92fe94d40448eb6dbd8536b4a2229 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/88a92fe94d40448eb6dbd8536b4a2229 2023-04-10 02:54:19,905 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/0147b5751c174844b4371001d9028737 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/0147b5751c174844b4371001d9028737 2023-04-10 02:54:19,906 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6c13c3e03b7f4729b02afa5d63ed0b92 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/6c13c3e03b7f4729b02afa5d63ed0b92 2023-04-10 02:54:19,911 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/decc60b4380b45b298d000a5788956fb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/939618c2269042788dd5b955022a9128, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/aab38db9ee724421ac61dd162876fa3b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/2b831db704014380ae446ac99b78393a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/56faf2c1bdd842d9971e2766faee2b93, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/4bb9c16123ae40dba44ac4e3652232b4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/b5b6e31ad7b947568d5028faac55cc20, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/a43bfcca62a6474bad18fdc0a49b39bd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e5a74b886555424e8320d04ff29d2528, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e3da6c5c20b84b11aa57c75a531d0ac0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/4976a657d95645718d249af43000ac08, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/d69ab199d099478c834df910c42e0698, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e6765b6aa96b49fa96af34297234fcb5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/91a10dd4daf04638aa6dff5b4c4cafc5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/0bb99e10d5c240998c690fea08d9c473, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/91f5854ef9984ba0b1af7fa6a846f85f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/7b427f992bf54ff0acc2448e6461cbff, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/547e4ee6b5164432878370720cf03ee5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/1acfca52020142189f2316a13c85b598, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/fd68f7ee926849fa8ee82f984c546782, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/0add03d448ce49a4a7f835b71a03dbb1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/fc72ce5325864607ac7342d471cc1f41, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/86c243c8db5e43cbae5f896778f0723d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/8fd14dc152f5482e82f72e020dba203d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/f8cac114940541ee9fc0f868a171393b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/de8c74b06c7a4733bd85a3f0343d259e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/86dc333fa1f84305a91a4f5c60426410, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/11b1aec82f724c6aa2a4bfab6ca14e4d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/5b9243b21d3c4e87b7e8efdf4e26b382, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/844c503a66ff431db13ed21e657c7e5f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/27a6ce81a1984088a057622bc24cb82c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/de40f81f673b465d92b5d8cfae4494f5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/d8266c5a26c0492ba602f3f00583c044, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/6adb1f8e517d4cd6a65d5b46344eecff, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/899e69458d1e4f78b3cc4cf0fb11b31c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/687e663948204bcd8abe8e326ae464d2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/44804488eed346f3891c4bc4da1bff9f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e11b7b67629f4dc4a8069696f2b4318a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/32b0194bf8f24f519ab00008b0d16745, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/6f7626207d354561b88b757deab32d55] to archive 2023-04-10 02:54:19,911 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:54:19,913 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/decc60b4380b45b298d000a5788956fb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/decc60b4380b45b298d000a5788956fb 2023-04-10 02:54:19,914 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/939618c2269042788dd5b955022a9128 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/939618c2269042788dd5b955022a9128 2023-04-10 02:54:19,915 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/aab38db9ee724421ac61dd162876fa3b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/aab38db9ee724421ac61dd162876fa3b 2023-04-10 02:54:19,917 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/2b831db704014380ae446ac99b78393a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/2b831db704014380ae446ac99b78393a 2023-04-10 02:54:19,918 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/56faf2c1bdd842d9971e2766faee2b93 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/56faf2c1bdd842d9971e2766faee2b93 2023-04-10 02:54:19,919 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/4bb9c16123ae40dba44ac4e3652232b4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/4bb9c16123ae40dba44ac4e3652232b4 2023-04-10 02:54:19,920 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/b5b6e31ad7b947568d5028faac55cc20 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/b5b6e31ad7b947568d5028faac55cc20 2023-04-10 02:54:19,921 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/a43bfcca62a6474bad18fdc0a49b39bd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/a43bfcca62a6474bad18fdc0a49b39bd 2023-04-10 02:54:19,922 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e5a74b886555424e8320d04ff29d2528 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e5a74b886555424e8320d04ff29d2528 2023-04-10 02:54:19,923 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e3da6c5c20b84b11aa57c75a531d0ac0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e3da6c5c20b84b11aa57c75a531d0ac0 2023-04-10 02:54:19,924 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/4976a657d95645718d249af43000ac08 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/4976a657d95645718d249af43000ac08 2023-04-10 02:54:19,925 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/d69ab199d099478c834df910c42e0698 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/d69ab199d099478c834df910c42e0698 2023-04-10 02:54:19,926 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e6765b6aa96b49fa96af34297234fcb5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e6765b6aa96b49fa96af34297234fcb5 2023-04-10 02:54:19,928 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/91a10dd4daf04638aa6dff5b4c4cafc5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/91a10dd4daf04638aa6dff5b4c4cafc5 2023-04-10 02:54:19,929 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/0bb99e10d5c240998c690fea08d9c473 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/0bb99e10d5c240998c690fea08d9c473 2023-04-10 02:54:19,930 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/91f5854ef9984ba0b1af7fa6a846f85f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/91f5854ef9984ba0b1af7fa6a846f85f 2023-04-10 02:54:19,931 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/7b427f992bf54ff0acc2448e6461cbff to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/7b427f992bf54ff0acc2448e6461cbff 2023-04-10 02:54:19,932 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/547e4ee6b5164432878370720cf03ee5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/547e4ee6b5164432878370720cf03ee5 2023-04-10 02:54:19,933 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/1acfca52020142189f2316a13c85b598 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/1acfca52020142189f2316a13c85b598 2023-04-10 02:54:19,934 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/fd68f7ee926849fa8ee82f984c546782 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/fd68f7ee926849fa8ee82f984c546782 2023-04-10 02:54:19,935 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/0add03d448ce49a4a7f835b71a03dbb1 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/0add03d448ce49a4a7f835b71a03dbb1 2023-04-10 02:54:19,936 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/fc72ce5325864607ac7342d471cc1f41 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/fc72ce5325864607ac7342d471cc1f41 2023-04-10 02:54:19,938 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/86c243c8db5e43cbae5f896778f0723d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/86c243c8db5e43cbae5f896778f0723d 2023-04-10 02:54:19,939 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/8fd14dc152f5482e82f72e020dba203d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/8fd14dc152f5482e82f72e020dba203d 2023-04-10 02:54:19,940 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/f8cac114940541ee9fc0f868a171393b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/f8cac114940541ee9fc0f868a171393b 2023-04-10 02:54:19,941 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/de8c74b06c7a4733bd85a3f0343d259e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/de8c74b06c7a4733bd85a3f0343d259e 2023-04-10 02:54:19,942 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/86dc333fa1f84305a91a4f5c60426410 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/86dc333fa1f84305a91a4f5c60426410 2023-04-10 02:54:19,943 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/11b1aec82f724c6aa2a4bfab6ca14e4d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/11b1aec82f724c6aa2a4bfab6ca14e4d 2023-04-10 02:54:19,944 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/5b9243b21d3c4e87b7e8efdf4e26b382 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/5b9243b21d3c4e87b7e8efdf4e26b382 2023-04-10 02:54:19,945 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/844c503a66ff431db13ed21e657c7e5f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/844c503a66ff431db13ed21e657c7e5f 2023-04-10 02:54:19,946 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/27a6ce81a1984088a057622bc24cb82c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/27a6ce81a1984088a057622bc24cb82c 2023-04-10 02:54:19,947 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/de40f81f673b465d92b5d8cfae4494f5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/de40f81f673b465d92b5d8cfae4494f5 2023-04-10 02:54:19,948 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/d8266c5a26c0492ba602f3f00583c044 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/d8266c5a26c0492ba602f3f00583c044 2023-04-10 02:54:19,949 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/6adb1f8e517d4cd6a65d5b46344eecff to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/6adb1f8e517d4cd6a65d5b46344eecff 2023-04-10 02:54:19,950 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/899e69458d1e4f78b3cc4cf0fb11b31c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/899e69458d1e4f78b3cc4cf0fb11b31c 2023-04-10 02:54:19,951 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/687e663948204bcd8abe8e326ae464d2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/687e663948204bcd8abe8e326ae464d2 2023-04-10 02:54:19,953 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/44804488eed346f3891c4bc4da1bff9f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/44804488eed346f3891c4bc4da1bff9f 2023-04-10 02:54:19,954 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e11b7b67629f4dc4a8069696f2b4318a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e11b7b67629f4dc4a8069696f2b4318a 2023-04-10 02:54:19,955 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/32b0194bf8f24f519ab00008b0d16745 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/32b0194bf8f24f519ab00008b0d16745 2023-04-10 02:54:19,957 DEBUG [StoreCloser-TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/6f7626207d354561b88b757deab32d55 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/6f7626207d354561b88b757deab32d55 2023-04-10 02:54:19,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/recovered.edits/751.seqid, newMaxSeqId=751, maxSeqId=1 2023-04-10 02:54:19,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595. 2023-04-10 02:54:19,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 66e59774c088f69ab4d8d1711037d595: 2023-04-10 02:54:19,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 66e59774c088f69ab4d8d1711037d595 2023-04-10 02:54:19,965 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=66e59774c088f69ab4d8d1711037d595, regionState=CLOSED 2023-04-10 02:54:19,965 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095259965"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095259965"}]},"ts":"1681095259965"} 2023-04-10 02:54:19,968 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-10 02:54:19,968 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 66e59774c088f69ab4d8d1711037d595, server=jenkins-hbase4.apache.org,38321,1681095222501 in 1.2520 sec 2023-04-10 02:54:19,970 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-10 02:54:19,970 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=66e59774c088f69ab4d8d1711037d595, UNASSIGN in 1.2580 sec 2023-04-10 02:54:19,970 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095259970"}]},"ts":"1681095259970"} 2023-04-10 02:54:19,971 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-10 02:54:19,973 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-10 02:54:19,975 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.2780 sec 2023-04-10 02:54:20,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-10 02:54:20,810 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-10 02:54:20,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-10 02:54:20,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:20,824 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:20,826 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:20,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-10 02:54:20,831 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595 2023-04-10 02:54:20,834 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/recovered.edits] 2023-04-10 02:54:20,837 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2b7d5689f9da488e82d0b04b9aa1df1c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/2b7d5689f9da488e82d0b04b9aa1df1c 2023-04-10 02:54:20,839 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/5c921888158049a88169647ab13aa203 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/A/5c921888158049a88169647ab13aa203 2023-04-10 02:54:20,842 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/1097ca66a7eb4ebfbeeb2e26adf2ef77 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/1097ca66a7eb4ebfbeeb2e26adf2ef77 2023-04-10 02:54:20,843 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/4ea93e94bf6847dc9f5901088ec7bea8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/B/4ea93e94bf6847dc9f5901088ec7bea8 2023-04-10 02:54:20,846 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/3cf551abfa2e4603a95c01935fd011c0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/3cf551abfa2e4603a95c01935fd011c0 2023-04-10 02:54:20,847 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e3f3c59df6434964a033aafcd72b8391 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/C/e3f3c59df6434964a033aafcd72b8391 2023-04-10 02:54:20,850 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/recovered.edits/751.seqid to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595/recovered.edits/751.seqid 2023-04-10 02:54:20,850 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/66e59774c088f69ab4d8d1711037d595 2023-04-10 02:54:20,850 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-10 02:54:20,853 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:20,873 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-10 02:54:20,921 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-10 02:54:20,923 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:20,923 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-10 02:54:20,923 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681095260923"}]},"ts":"9223372036854775807"} 2023-04-10 02:54:20,925 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-10 02:54:20,925 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 66e59774c088f69ab4d8d1711037d595, NAME => 'TestAcidGuarantees,,1681095225549.66e59774c088f69ab4d8d1711037d595.', STARTKEY => '', ENDKEY => ''}] 2023-04-10 02:54:20,925 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-10 02:54:20,926 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681095260925"}]},"ts":"9223372036854775807"} 2023-04-10 02:54:20,927 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-10 02:54:20,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-10 02:54:20,930 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:20,931 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 113 msec 2023-04-10 02:54:21,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-10 02:54:21,131 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-10 02:54:21,147 INFO [Listener at localhost/44533] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=231 (was 208) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x10f63bbb-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x10f63bbb-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-119807682_15 at /127.0.0.1:54574 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x10f63bbb-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x10f63bbb-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:38321-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=542 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=334 (was 407), ProcessCount=184 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=1734 (was 2439) 2023-04-10 02:54:21,158 INFO [Listener at localhost/44533] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=231, OpenFileDescriptor=542, MaxFileDescriptor=60000, SystemLoadAverage=334, ProcessCount=184, AvailableMemoryMB=1734 2023-04-10 02:54:21,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-10 02:54:21,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-10 02:54:21,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:21,163 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-10 02:54:21,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-10 02:54:21,164 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-10 02:54:21,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-10 02:54:21,165 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:21,166 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc empty. 2023-04-10 02:54:21,166 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:21,166 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-10 02:54:21,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-10 02:54:21,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-10 02:54:21,579 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-10 02:54:21,580 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a944754ff766fcb82f3c238c462e40cc, NAME => 'TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp 2023-04-10 02:54:21,586 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:54:21,586 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a944754ff766fcb82f3c238c462e40cc, disabling compactions & flushes 2023-04-10 02:54:21,586 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:21,586 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:21,586 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. after waiting 0 ms 2023-04-10 02:54:21,586 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:21,587 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:21,587 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:21,589 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-10 02:54:21,590 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095261589"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095261589"}]},"ts":"1681095261589"} 2023-04-10 02:54:21,591 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-10 02:54:21,592 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-10 02:54:21,592 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095261592"}]},"ts":"1681095261592"} 2023-04-10 02:54:21,593 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-10 02:54:21,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a944754ff766fcb82f3c238c462e40cc, ASSIGN}] 2023-04-10 02:54:21,598 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a944754ff766fcb82f3c238c462e40cc, ASSIGN 2023-04-10 02:54:21,598 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a944754ff766fcb82f3c238c462e40cc, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38321,1681095222501; forceNewPlan=false, retain=false 2023-04-10 02:54:21,749 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=a944754ff766fcb82f3c238c462e40cc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:21,750 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095261749"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095261749"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095261749"}]},"ts":"1681095261749"} 2023-04-10 02:54:21,751 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:54:21,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-10 02:54:21,908 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:21,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a944754ff766fcb82f3c238c462e40cc, NAME => 'TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.', STARTKEY => '', ENDKEY => ''} 2023-04-10 02:54:21,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:21,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:54:21,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:21,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:21,909 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:21,911 DEBUG [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A 2023-04-10 02:54:21,911 DEBUG [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A 2023-04-10 02:54:21,911 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:54:21,911 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a944754ff766fcb82f3c238c462e40cc columnFamilyName A 2023-04-10 02:54:21,912 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.HStore(310): Store=a944754ff766fcb82f3c238c462e40cc/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:54:21,912 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:21,913 DEBUG [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B 2023-04-10 02:54:21,913 DEBUG [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B 2023-04-10 02:54:21,913 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:54:21,913 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a944754ff766fcb82f3c238c462e40cc columnFamilyName B 2023-04-10 02:54:21,914 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.HStore(310): Store=a944754ff766fcb82f3c238c462e40cc/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:54:21,914 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:21,915 DEBUG [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C 2023-04-10 02:54:21,915 DEBUG [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C 2023-04-10 02:54:21,915 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:54:21,915 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a944754ff766fcb82f3c238c462e40cc columnFamilyName C 2023-04-10 02:54:21,916 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.HStore(310): Store=a944754ff766fcb82f3c238c462e40cc/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:54:21,916 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:21,917 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:21,917 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:21,919 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-10 02:54:21,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:21,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-10 02:54:21,922 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a944754ff766fcb82f3c238c462e40cc; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60255180, jitterRate=-0.10212785005569458}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-10 02:54:21,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:21,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., pid=18, masterSystemTime=1681095261903 2023-04-10 02:54:21,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:21,924 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:21,925 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=a944754ff766fcb82f3c238c462e40cc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:21,925 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095261924"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681095261924"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681095261924"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681095261924"}]},"ts":"1681095261924"} 2023-04-10 02:54:21,927 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-10 02:54:21,927 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 in 175 msec 2023-04-10 02:54:21,929 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-10 02:54:21,929 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a944754ff766fcb82f3c238c462e40cc, ASSIGN in 331 msec 2023-04-10 02:54:21,929 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-10 02:54:21,930 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095261930"}]},"ts":"1681095261930"} 2023-04-10 02:54:21,930 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-10 02:54:21,933 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-10 02:54:21,934 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 772 msec 2023-04-10 02:54:22,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-10 02:54:22,268 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-10 02:54:22,269 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x33c943f0 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:22,274 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b8d0eb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:22,276 DEBUG [Listener at localhost/44533] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:22,279 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53850, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:22,281 DEBUG [Listener at localhost/44533] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-10 02:54:22,286 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58252, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-10 02:54:22,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-10 02:54:22,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-10 02:54:22,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:22,718 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-10 02:54:22,719 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-10 02:54:22,719 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-10 02:54:22,723 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-10 02:54:22,734 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a944754ff766fcb82f3c238c462e40cc, REOPEN/MOVE}] 2023-04-10 02:54:22,735 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a944754ff766fcb82f3c238c462e40cc, REOPEN/MOVE 2023-04-10 02:54:22,736 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=a944754ff766fcb82f3c238c462e40cc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:22,736 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095262736"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095262736"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095262736"}]},"ts":"1681095262736"} 2023-04-10 02:54:22,737 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:54:22,889 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:22,890 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a944754ff766fcb82f3c238c462e40cc, disabling compactions & flushes 2023-04-10 02:54:22,890 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:22,890 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:22,890 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. after waiting 0 ms 2023-04-10 02:54:22,890 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:22,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-10 02:54:22,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:22,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:22,894 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: a944754ff766fcb82f3c238c462e40cc to self. 2023-04-10 02:54:22,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:22,896 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=a944754ff766fcb82f3c238c462e40cc, regionState=CLOSED 2023-04-10 02:54:22,897 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095262896"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095262896"}]},"ts":"1681095262896"} 2023-04-10 02:54:22,899 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-10 02:54:22,899 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 in 161 msec 2023-04-10 02:54:22,900 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a944754ff766fcb82f3c238c462e40cc, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,38321,1681095222501; forceNewPlan=false, retain=true 2023-04-10 02:54:23,050 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=a944754ff766fcb82f3c238c462e40cc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,051 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095263050"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095263050"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095263050"}]},"ts":"1681095263050"} 2023-04-10 02:54:23,052 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:54:23,207 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:23,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a944754ff766fcb82f3c238c462e40cc, NAME => 'TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.', STARTKEY => '', ENDKEY => ''} 2023-04-10 02:54:23,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:23,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:54:23,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:23,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:23,210 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:23,211 DEBUG [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A 2023-04-10 02:54:23,212 DEBUG [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A 2023-04-10 02:54:23,212 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:54:23,214 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a944754ff766fcb82f3c238c462e40cc columnFamilyName A 2023-04-10 02:54:23,216 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.HStore(310): Store=a944754ff766fcb82f3c238c462e40cc/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:54:23,217 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:23,218 DEBUG [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B 2023-04-10 02:54:23,218 DEBUG [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B 2023-04-10 02:54:23,218 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:54:23,218 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a944754ff766fcb82f3c238c462e40cc columnFamilyName B 2023-04-10 02:54:23,219 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.HStore(310): Store=a944754ff766fcb82f3c238c462e40cc/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:54:23,219 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:23,220 DEBUG [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C 2023-04-10 02:54:23,220 DEBUG [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C 2023-04-10 02:54:23,220 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:54:23,220 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a944754ff766fcb82f3c238c462e40cc columnFamilyName C 2023-04-10 02:54:23,221 INFO [StoreOpener-a944754ff766fcb82f3c238c462e40cc-1] regionserver.HStore(310): Store=a944754ff766fcb82f3c238c462e40cc/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:54:23,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:23,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:23,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:23,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-10 02:54:23,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:23,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a944754ff766fcb82f3c238c462e40cc; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59705026, jitterRate=-0.11032578349113464}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-10 02:54:23,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:23,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., pid=23, masterSystemTime=1681095263204 2023-04-10 02:54:23,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:23,229 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:23,230 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=a944754ff766fcb82f3c238c462e40cc, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,230 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095263229"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681095263229"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681095263229"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681095263229"}]},"ts":"1681095263229"} 2023-04-10 02:54:23,233 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-10 02:54:23,233 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 in 179 msec 2023-04-10 02:54:23,234 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-10 02:54:23,234 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a944754ff766fcb82f3c238c462e40cc, REOPEN/MOVE in 499 msec 2023-04-10 02:54:23,237 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-10 02:54:23,237 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 513 msec 2023-04-10 02:54:23,239 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 939 msec 2023-04-10 02:54:23,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-10 02:54:23,250 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x5b3934fb to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:23,254 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34135972, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:23,255 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x1aa5eef5 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:23,259 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a71edf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:23,259 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x1441c9f0 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:23,263 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21265fcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:23,264 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x69f15cb4 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:23,270 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7aba8e46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:23,270 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x6c1c92c6 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:23,274 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@feb770d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:23,274 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x54451f11 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:23,279 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c2598b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:23,280 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:23,285 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:23,286 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:23,292 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:23,292 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3bb940 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:23,296 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e27a32, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:23,304 DEBUG [hconnection-0xf5e0ee1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:23,308 DEBUG [hconnection-0x22904621-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:23,308 DEBUG [hconnection-0x8f755f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:23,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:23,310 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53856, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:23,312 DEBUG [hconnection-0x4c496953-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:23,313 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53878, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:23,313 DEBUG [hconnection-0x6fa4f6cd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:23,313 DEBUG [hconnection-0x74286642-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:23,315 DEBUG [hconnection-0x4582e14f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:23,315 DEBUG [hconnection-0x57102b15-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:23,316 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53902, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:23,318 DEBUG [hconnection-0x649a55e5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:23,322 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53862, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:23,322 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53860, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:23,322 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53894, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:23,322 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53892, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:23,322 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53914, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:23,323 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53910, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:23,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:23,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:23,331 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:23,331 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:23,331 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:23,331 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:23,331 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,331 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,334 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,334 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:23,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:23,335 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,335 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:23,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:23,345 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:54:23,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:23,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:23,345 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:23,345 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:23,345 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:23,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:23,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:23,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:23,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:23,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:23,347 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:23,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:23,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:23,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:23,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:23,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:23,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:23,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:23,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:23,371 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,371 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:23,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:23,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f5674a2[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:23,374 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:23,374 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,376 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,376 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:23,376 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:23,376 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:23,376 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,376 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:23,378 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:23,378 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:23,378 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:23,378 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:23,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095323369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095323371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095323373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095323383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:23,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095323384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,384 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:23,384 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:23,384 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:23,384 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:23,384 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:23,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:23,387 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,387 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,387 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:23,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:23,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:23,388 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:23,390 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:23,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:23,393 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:23,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:23,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:23,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:23,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:23,393 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:23,393 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:23,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:23,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:23,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095323483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095323483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095323484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095323486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095323486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,493 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:23,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:23,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:23,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:23,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:23,502 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:23,502 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:23,502 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:23,502 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:23,503 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,503 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,505 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:23,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:23,506 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,506 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:23,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:23,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:23,512 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:23,512 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:23,512 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:23,513 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:23,513 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:23,513 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:23,513 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:23,518 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:23,520 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:23,521 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,521 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:23,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:23,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18b5d5e7[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:23,522 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:23,522 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,523 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,524 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:23,524 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:23,524 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:23,524 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,525 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:23,528 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:23,528 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:23,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,528 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:23,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:23,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:23,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:23,532 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:23,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:23,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:23,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:23,532 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:23,535 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,535 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,535 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:23,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:23,536 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,544 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:23,544 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:23,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:23,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:23,544 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,544 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,544 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,544 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,544 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,545 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,545 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,545 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:23,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:23,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:23,545 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:23,545 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:23,645 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:23,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:23,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:23,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:23,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:23,660 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:23,660 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:23,660 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:23,660 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:23,661 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,661 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,664 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:23,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:23,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:23,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:23,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:23,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:23,684 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:23,684 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:23,684 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:23,684 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:23,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095323687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095323687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095323687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095323689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095323689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,694 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,694 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:23,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@612dab69[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:23,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:23,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,698 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:23,698 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:23,698 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:23,698 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,698 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:23,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:23,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:23,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:23,699 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:23,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:23,703 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:23,703 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:23,703 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:23,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:23,704 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:23,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:23,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:23,708 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,709 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,712 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:23,712 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:23,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:23,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:23,713 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,713 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,713 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,713 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,713 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:23,713 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:23,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:23,713 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,713 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,713 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:23,713 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:23,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:23,813 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:23,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:23,817 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041069aff50aa3ef432cb60bd72b70fdeb9c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041069aff50aa3ef432cb60bd72b70fdeb9c 2023-04-10 02:54:23,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:23,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:23,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:23,832 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:23,832 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:23,832 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:23,832 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:23,833 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,833 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,834 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,834 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:23,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:23,835 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,835 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:23,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:23,844 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:23,844 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:23,844 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:23,844 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:23,844 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:23,844 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:23,844 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:23,844 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:23,844 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:23,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:23,846 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@428b2565[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:23,849 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:23,849 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,850 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:23,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:23,851 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,851 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:23,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:23,853 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:23,853 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:23,853 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:23,854 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:23,857 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:23,857 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:23,857 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:23,857 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:23,858 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:23,859 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,859 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:23,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,859 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:23,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:23,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,863 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:23,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:23,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,867 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:23,867 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,867 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:23,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:23,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:23,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:23,867 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:23,867 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:23,867 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:23,867 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:23,867 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:23,867 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,867 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:23,867 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:23,868 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:23,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:23,967 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:23,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:23,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:23,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095323990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095323991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095323993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095323993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:23,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:23,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095323998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:24,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:24,008 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:24,008 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:24,008 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:24,008 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:24,009 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,009 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,010 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:24,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:24,010 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,011 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,011 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:24,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:24,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:24,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:24,019 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:24,019 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:24,019 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:24,020 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:24,020 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:24,020 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:24,020 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:24,023 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,023 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:24,023 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:24,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:24,025 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:24,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e24aaf2[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:24,025 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,027 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:24,027 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,027 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:24,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:24,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:24,033 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:24,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:24,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:24,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:24,034 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:24,035 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,035 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:24,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,036 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:24,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,038 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:24,041 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:24,041 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:24,041 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:24,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:24,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:24,041 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:24,041 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,041 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,041 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:24,066 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:24,067 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:24,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:24,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:24,167 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:24,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:24,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:24,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:24,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:24,174 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:24,174 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:24,174 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:24,174 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:24,174 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,174 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,175 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:24,176 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:24,176 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,176 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:24,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:24,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:24,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:24,180 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:24,180 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:24,180 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:24,180 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:24,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:24,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:24,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:24,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:24,183 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:24,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:24,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:24,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:24,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28084e3c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:24,185 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:24,185 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,186 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,186 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:24,186 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:24,186 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:24,186 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,186 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:24,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:24,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:24,193 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:24,193 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:24,193 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:24,193 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:24,193 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:24,195 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,195 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:24,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,195 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:24,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:24,195 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,197 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,199 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,199 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,199 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,199 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:24,199 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,200 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,200 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,200 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:24,200 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:24,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:24,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:24,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:24,200 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:24,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:24,200 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:24,200 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:24,242 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/2f1a2c282e2643a3af0b9a3b1d4e3902 2023-04-10 02:54:24,264 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/6df9067e043e49a5888b5a77b27fbb1e 2023-04-10 02:54:24,286 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/259c48e4c3a649d1b9aa2da203082bee 2023-04-10 02:54:24,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/2f1a2c282e2643a3af0b9a3b1d4e3902 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/2f1a2c282e2643a3af0b9a3b1d4e3902 2023-04-10 02:54:24,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/2f1a2c282e2643a3af0b9a3b1d4e3902, entries=150, sequenceid=17, filesize=25.1 K 2023-04-10 02:54:24,300 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:24,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/6df9067e043e49a5888b5a77b27fbb1e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/6df9067e043e49a5888b5a77b27fbb1e 2023-04-10 02:54:24,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:24,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:24,308 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/6df9067e043e49a5888b5a77b27fbb1e, entries=150, sequenceid=17, filesize=11.6 K 2023-04-10 02:54:24,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/259c48e4c3a649d1b9aa2da203082bee as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/259c48e4c3a649d1b9aa2da203082bee 2023-04-10 02:54:24,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:24,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:24,316 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:24,316 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:24,316 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:24,316 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:24,317 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/259c48e4c3a649d1b9aa2da203082bee, entries=150, sequenceid=17, filesize=11.6 K 2023-04-10 02:54:24,318 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,318 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,319 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for a944754ff766fcb82f3c238c462e40cc in 973ms, sequenceid=17, compaction requested=false 2023-04-10 02:54:24,319 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-10 02:54:24,320 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:24,320 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:24,320 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:24,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,332 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,332 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:24,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:24,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:24,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:24,336 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:24,336 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:24,336 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:24,344 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:24,344 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:24,344 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-10 02:54:24,345 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:24,345 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:24,345 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:24,345 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:24,345 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:24,345 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:24,395 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041025a31c6f36a04c6d95f3e526403d0b08 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041025a31c6f36a04c6d95f3e526403d0b08 2023-04-10 02:54:24,412 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/c943a7bb010d490b947669e76977f91b 2023-04-10 02:54:24,456 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/eaa106e1299a46d1966ab13df029cb28 2023-04-10 02:54:24,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:24,496 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:24,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095324505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095324509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095324539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095324539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095324539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095324640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095324642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095324642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095324642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095324651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095324843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095324844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095324844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095324847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:24,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095324852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,882 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/5cdaa54fa5c94479a4095b5a2416e8c6 2023-04-10 02:54:24,887 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/c943a7bb010d490b947669e76977f91b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/c943a7bb010d490b947669e76977f91b 2023-04-10 02:54:24,894 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/c943a7bb010d490b947669e76977f91b, entries=150, sequenceid=37, filesize=25.1 K 2023-04-10 02:54:24,895 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/eaa106e1299a46d1966ab13df029cb28 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/eaa106e1299a46d1966ab13df029cb28 2023-04-10 02:54:24,900 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/eaa106e1299a46d1966ab13df029cb28, entries=150, sequenceid=37, filesize=11.6 K 2023-04-10 02:54:24,901 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/5cdaa54fa5c94479a4095b5a2416e8c6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/5cdaa54fa5c94479a4095b5a2416e8c6 2023-04-10 02:54:24,906 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/5cdaa54fa5c94479a4095b5a2416e8c6, entries=150, sequenceid=37, filesize=11.6 K 2023-04-10 02:54:24,907 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for a944754ff766fcb82f3c238c462e40cc in 563ms, sequenceid=37, compaction requested=false 2023-04-10 02:54:24,908 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:24,908 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:24,911 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,911 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:24,912 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,912 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:24,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:24,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2472ceb7[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:24,914 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:24,914 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,917 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,917 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:24,917 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:24,917 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:24,917 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,917 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:24,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:24,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:24,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:24,920 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:24,927 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:24,927 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:24,927 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:24,927 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:24,927 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:24,929 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:24,930 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,931 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,934 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,934 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,934 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:24,934 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:24,934 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,934 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:24,934 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:24,934 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:24,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:24,934 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:24,935 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:24,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:24,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:24,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:24,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:24,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:24,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:24,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:24,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:25,035 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:25,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:25,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:25,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:25,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:25,039 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:25,039 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:25,039 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:25,040 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:25,040 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:25,040 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:25,043 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,043 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:25,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:25,043 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,043 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:25,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:25,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:25,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:25,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:25,045 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:25,045 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:25,046 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-10 02:54:25,046 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:25,046 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:25,046 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:25,046 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:25,046 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:25,046 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:25,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:25,148 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:25,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095325158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095325159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095325160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095325161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095325164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095325264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095325264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095325265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095325265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095325267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,466 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304107aced8f9be614bd6a2639cdf7b9e6947 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304107aced8f9be614bd6a2639cdf7b9e6947 2023-04-10 02:54:25,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095325467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095325468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095325468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095325469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095325472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,476 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/e4952c6765614d629fc9be85dd4baf67 2023-04-10 02:54:25,506 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-10 02:54:25,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095325769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095325771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095325772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095325773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:25,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095325775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,888 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/058bda01b35741dfa2e0c3e22c71533b 2023-04-10 02:54:25,900 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/20ba4388b40f46509623cc0a316ca48f 2023-04-10 02:54:25,905 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/e4952c6765614d629fc9be85dd4baf67 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/e4952c6765614d629fc9be85dd4baf67 2023-04-10 02:54:25,910 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/e4952c6765614d629fc9be85dd4baf67, entries=150, sequenceid=50, filesize=25.1 K 2023-04-10 02:54:25,911 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/058bda01b35741dfa2e0c3e22c71533b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/058bda01b35741dfa2e0c3e22c71533b 2023-04-10 02:54:25,918 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/058bda01b35741dfa2e0c3e22c71533b, entries=150, sequenceid=50, filesize=11.6 K 2023-04-10 02:54:25,920 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/20ba4388b40f46509623cc0a316ca48f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/20ba4388b40f46509623cc0a316ca48f 2023-04-10 02:54:25,927 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/20ba4388b40f46509623cc0a316ca48f, entries=150, sequenceid=50, filesize=11.6 K 2023-04-10 02:54:25,928 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for a944754ff766fcb82f3c238c462e40cc in 882ms, sequenceid=50, compaction requested=true 2023-04-10 02:54:25,929 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:25,929 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:25,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:25,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:25,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:25,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:25,933 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:25,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:25,934 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:25,934 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:25,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:25,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f497751[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:25,936 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:25,936 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:25,938 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:25,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:25,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:25,939 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:25,939 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:25,939 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,939 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:25,939 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:25,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:25,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:25,943 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:25,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:25,945 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:25,945 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:25,945 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:25,945 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:25,945 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:25,947 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:25,947 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:25,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:25,947 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:25,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:25,949 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:25,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:25,953 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,953 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:25,953 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:25,953 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:25,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:25,953 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:25,953 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:25,953 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:25,953 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:25,953 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:25,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:25,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:25,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:25,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:25,954 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:25,954 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:25,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:26,054 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:26,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:26,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:26,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:26,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:26,060 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:26,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:26,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:26,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:26,061 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:26,061 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:26,063 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:26,063 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:26,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:26,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:26,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:26,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:26,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:26,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:26,065 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:26,065 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:26,065 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:26,066 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:26,066 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:26,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:26,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:26,066 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-10 02:54:26,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:26,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:26,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:26,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:26,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:26,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:26,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:26,274 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:26,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095326280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095326282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095326283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095326284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095326284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095326385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095326385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095326386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095326390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095326390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,524 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304103c2e70a862ea42ad9ea1fc4049826fbb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304103c2e70a862ea42ad9ea1fc4049826fbb 2023-04-10 02:54:26,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095326588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095326588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095326589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095326597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095326605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095326891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095326892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095326893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095326901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:26,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095326910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:26,930 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/26e508a7ec2a4823abd248d565e5eaaa 2023-04-10 02:54:26,946 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/985ab9a5fb2e4bf886161c356e519108 2023-04-10 02:54:26,956 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/3595ff077d144d0a9108495086d0cad0 2023-04-10 02:54:26,972 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/26e508a7ec2a4823abd248d565e5eaaa as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/26e508a7ec2a4823abd248d565e5eaaa 2023-04-10 02:54:26,980 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/26e508a7ec2a4823abd248d565e5eaaa, entries=150, sequenceid=69, filesize=25.1 K 2023-04-10 02:54:26,981 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/985ab9a5fb2e4bf886161c356e519108 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/985ab9a5fb2e4bf886161c356e519108 2023-04-10 02:54:26,987 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/985ab9a5fb2e4bf886161c356e519108, entries=150, sequenceid=69, filesize=11.6 K 2023-04-10 02:54:26,990 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/3595ff077d144d0a9108495086d0cad0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/3595ff077d144d0a9108495086d0cad0 2023-04-10 02:54:27,004 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/3595ff077d144d0a9108495086d0cad0, entries=150, sequenceid=69, filesize=11.6 K 2023-04-10 02:54:27,005 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for a944754ff766fcb82f3c238c462e40cc in 939ms, sequenceid=69, compaction requested=true 2023-04-10 02:54:27,005 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:27,005 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:27,005 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:27,005 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:27,005 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,005 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:27,005 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:27,008 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,008 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:27,008 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:27,008 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:27,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:27,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4813970[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:27,010 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:27,011 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:27,012 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:27,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:27,012 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,012 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:27,014 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:27,016 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:27,017 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:27,017 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:27,017 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:27,017 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:27,023 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:27,023 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:27,023 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:27,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:27,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:27,024 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,026 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,031 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:27,031 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:27,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:27,031 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:27,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:27,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:27,031 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,031 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:27,031 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:27,031 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,031 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:27,031 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:27,031 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:27,031 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:27,031 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:27,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:27,132 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:27,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:27,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:27,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:27,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:27,136 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:27,136 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:27,136 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:27,136 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:27,136 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:27,136 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:27,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:27,137 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:27,138 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,138 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:27,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:27,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:27,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:27,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:27,140 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-10 02:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:27,156 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304109c1a16af986e4b95a9082e751af779f7 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304109c1a16af986e4b95a9082e751af779f7 2023-04-10 02:54:27,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:27,398 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:27,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:27,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095327415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:27,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095327415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:27,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095327416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:27,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095327416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:27,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095327416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:27,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095327518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:27,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095327522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:27,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095327522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:27,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095327523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,562 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/51a235e84cef45449ae69d492fbdb4dd 2023-04-10 02:54:27,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:27,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095327726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:27,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095327727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:27,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095327727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:27,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095327728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:27,976 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/bc6c1919b19943d98c7ead72f4618e70 2023-04-10 02:54:27,993 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/c5c96b26056b4130a418f78e2d4db46b 2023-04-10 02:54:27,999 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/51a235e84cef45449ae69d492fbdb4dd as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/51a235e84cef45449ae69d492fbdb4dd 2023-04-10 02:54:28,004 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/51a235e84cef45449ae69d492fbdb4dd, entries=150, sequenceid=83, filesize=25.1 K 2023-04-10 02:54:28,005 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/bc6c1919b19943d98c7ead72f4618e70 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/bc6c1919b19943d98c7ead72f4618e70 2023-04-10 02:54:28,010 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/bc6c1919b19943d98c7ead72f4618e70, entries=150, sequenceid=83, filesize=11.6 K 2023-04-10 02:54:28,011 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/c5c96b26056b4130a418f78e2d4db46b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/c5c96b26056b4130a418f78e2d4db46b 2023-04-10 02:54:28,018 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/c5c96b26056b4130a418f78e2d4db46b, entries=150, sequenceid=83, filesize=11.6 K 2023-04-10 02:54:28,019 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for a944754ff766fcb82f3c238c462e40cc in 879ms, sequenceid=83, compaction requested=true 2023-04-10 02:54:28,019 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:28,019 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,019 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:28,019 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:28,019 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,019 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:28,019 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:28,021 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,021 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,021 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,021 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:28,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e12fe72[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:28,023 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:28,023 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,024 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,024 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,024 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:28,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:28,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:28,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:28,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:28,035 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:28,035 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,035 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:28,035 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,035 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-10 02:54:28,035 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:28,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:28,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:28,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:28,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:28,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:28,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:28,041 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:28,041 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:28,041 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:28,041 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:28,041 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:28,050 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,050 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,050 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095328046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,056 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095328050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095328051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095328052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,057 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:28,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,059 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,060 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:28,060 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,060 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,060 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,060 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,060 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:28,060 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:28,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095328153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095328157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,160 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:28,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095328158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095328158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:28,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:28,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:28,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:28,164 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:28,164 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,164 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:28,164 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:28,165 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,165 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,166 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,166 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,166 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,166 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:28,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:28,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:28,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:28,167 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:28,167 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,167 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:28,168 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,168 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:28,168 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:28,168 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:28,169 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,170 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,170 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,170 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:28,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c278271[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:28,171 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:28,171 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,173 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:28,173 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:28,173 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:28,173 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,173 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:28,174 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:28,174 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,174 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,174 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:28,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:28,177 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:28,177 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:28,177 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:28,177 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:28,177 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:28,178 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,178 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,178 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,179 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,180 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,183 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,183 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,183 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,183 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,183 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,183 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:28,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:28,183 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,183 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,183 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,183 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:28,183 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:28,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,183 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,283 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:28,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:28,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:28,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:28,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:28,287 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:28,287 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,287 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:28,287 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:28,288 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,288 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,290 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,291 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,291 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:28,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:28,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:28,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:28,292 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,292 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:28,292 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:28,292 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:28,294 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,294 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,294 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,294 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:28,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28a0855b[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:28,295 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:28,295 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,296 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,296 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:28,297 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:28,297 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:28,297 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,297 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:28,298 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,298 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:28,298 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,298 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:28,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:28,300 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:28,300 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:28,300 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:28,300 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:28,300 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:28,303 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,303 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,304 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,305 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,307 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:28,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:28,307 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:28,307 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:28,308 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,308 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,308 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,308 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,308 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,308 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,308 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,308 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,310 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,310 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:28,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095328356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095328361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095328362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095328362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,408 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:28,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:28,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:28,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:28,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:28,412 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:28,412 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,412 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:28,412 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:28,412 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,412 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,413 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,414 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,414 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:28,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:28,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:28,415 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:28,415 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,415 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:28,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:28,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:28,416 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,416 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:28,416 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:28,416 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:28,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:28,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:28,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:28,419 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,419 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,419 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,419 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:28,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25677386[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:28,421 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:28,421 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,422 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:28,422 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,422 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:28,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:28,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:28,426 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:28,426 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:28,426 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:28,426 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:28,426 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:28,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,428 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,430 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095328429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,434 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,434 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,434 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:28,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:28,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:28,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,434 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,434 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,434 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:28,434 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:28,456 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304107b6ce678278844a4ad7ccdfcaf7f687e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304107b6ce678278844a4ad7ccdfcaf7f687e 2023-04-10 02:54:28,466 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=102, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/98d5ed57ccee439db0667428fe6c9665 2023-04-10 02:54:28,534 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:28,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:28,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:28,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:28,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:28,538 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:28,538 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,538 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:28,538 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:28,539 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,539 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,540 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:28,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:28,541 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:28,542 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,542 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:28,542 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:28,542 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,544 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:28,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c7fe836[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:28,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:28,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,548 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:28,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:28,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:28,548 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,548 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:28,549 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,549 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:28,549 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,549 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:28,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:28,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:28,552 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:28,552 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:28,552 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:28,552 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:28,552 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:28,553 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,553 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,553 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,554 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,555 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,557 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,557 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,557 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:28,557 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,557 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:28,557 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,557 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,558 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,558 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,558 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,558 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:28,558 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:28,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,658 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:28,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:28,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:28,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:28,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:28,661 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:28,661 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,661 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:28,661 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:28,662 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,662 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,663 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,663 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095328659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:28,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:28,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:28,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:28,665 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:28,665 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,665 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:28,665 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:28,665 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:28,665 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,665 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:28,666 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:28,666 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,666 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:28,666 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:28,666 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,666 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:28,666 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:28,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095328666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095328666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,668 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:28,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095328668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:28,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c2bddc2[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:28,670 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:28,670 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,671 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,671 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:28,671 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:28,671 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,671 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:28,671 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:28,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:28,674 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:28,676 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:28,676 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:28,677 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:28,677 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:28,677 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:28,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,680 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,681 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,686 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,686 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,686 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,686 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,686 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:28,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,686 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:28,686 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:28,686 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,687 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:28,687 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,687 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,687 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,787 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:28,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:28,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:28,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:28,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:28,790 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:28,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:28,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:28,791 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,791 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,793 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,793 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,794 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,794 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:28,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:28,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:28,795 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,795 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:28,795 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:28,795 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:28,797 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:28,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@108d0d41[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:28,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:28,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,800 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,800 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:28,800 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:28,800 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:28,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:28,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,801 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:28,801 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,801 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:28,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:28,804 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:28,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:28,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:28,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:28,804 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:28,809 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,810 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,810 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,811 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,814 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:28,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:28,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:28,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,815 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,815 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,815 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:28,815 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:28,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,878 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/c428dc41a1834d909a831b0c66b481a0 2023-04-10 02:54:28,915 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:28,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:28,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:28,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:28,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:28,918 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:28,918 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,918 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:28,918 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:28,919 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,919 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,920 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,920 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,920 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,920 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:28,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:28,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:28,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:28,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:28,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:28,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:28,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:28,922 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,922 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:28,922 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:28,922 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:28,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:28,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:28,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:28,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:28,924 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,924 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,924 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,924 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:28,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ee8b795[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:28,925 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:28,925 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,927 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,928 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:28,928 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:28,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:28,928 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:28,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:28,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,929 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:28,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:28,932 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:28,932 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:28,932 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:28,932 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:28,932 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:28,934 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,934 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,935 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:28,935 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,938 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,938 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,938 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:28,938 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:28,938 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:28,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:28,939 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:28,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:28,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:28,939 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:28,939 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,939 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:28,939 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:28,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:28,939 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:28,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:28,939 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:29,039 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:29,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:29,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:29,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:29,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:29,044 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:29,044 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:29,044 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:29,044 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:29,044 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,044 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,045 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:29,045 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:29,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:29,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:29,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:29,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:29,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:29,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:29,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:29,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:29,048 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:29,048 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:29,048 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:29,048 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:29,051 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:29,052 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:29,052 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,052 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:29,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:29,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:29,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:29,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d75c026[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:29,054 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:29,054 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,055 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:29,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:29,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:29,056 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,056 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:29,057 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:29,064 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:29,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:29,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:29,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:29,064 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:29,066 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,066 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:29,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,066 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:29,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,066 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,068 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,070 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,070 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,070 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,070 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,070 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,070 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,070 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,070 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:29,070 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:29,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:29,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:29,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:29,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:29,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:29,070 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:29,071 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:29,071 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:29,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:29,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095329167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:29,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095329168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,171 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:29,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:29,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:29,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095329170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:29,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:29,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:29,174 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:29,174 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:29,175 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:29,175 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:29,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:29,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095329173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,176 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,176 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,179 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:29,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:29,179 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:29,180 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,180 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:29,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:29,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:29,181 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:29,181 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:29,181 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:29,184 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:29,184 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:29,184 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:29,184 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,188 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:29,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:29,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fd2cd87[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:29,190 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:29,190 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,191 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:29,191 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:29,191 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:29,193 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:29,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:29,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:29,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:29,196 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:29,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:29,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:29,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:29,196 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:29,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:29,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:29,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:29,198 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,200 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:29,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,202 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:29,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:29,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:29,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:29,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:29,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,203 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,203 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:29,203 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:29,203 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:29,295 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/d71a3d8b28ca45658a076a134c4f9d63 2023-04-10 02:54:29,300 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/98d5ed57ccee439db0667428fe6c9665 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/98d5ed57ccee439db0667428fe6c9665 2023-04-10 02:54:29,303 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:29,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:29,304 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/98d5ed57ccee439db0667428fe6c9665, entries=200, sequenceid=102, filesize=31.9 K 2023-04-10 02:54:29,305 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/c428dc41a1834d909a831b0c66b481a0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/c428dc41a1834d909a831b0c66b481a0 2023-04-10 02:54:29,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:29,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:29,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:29,308 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:29,308 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:29,308 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:29,308 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:29,309 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,309 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,311 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:29,311 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:29,311 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,311 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:29,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:29,313 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:29,313 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:29,313 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:29,313 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:29,314 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/c428dc41a1834d909a831b0c66b481a0, entries=150, sequenceid=102, filesize=11.6 K 2023-04-10 02:54:29,315 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/d71a3d8b28ca45658a076a134c4f9d63 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/d71a3d8b28ca45658a076a134c4f9d63 2023-04-10 02:54:29,317 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/d71a3d8b28ca45658a076a134c4f9d63, entries=150, sequenceid=102, filesize=11.6 K 2023-04-10 02:54:29,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for a944754ff766fcb82f3c238c462e40cc in 1285ms, sequenceid=102, compaction requested=true 2023-04-10 02:54:29,321 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:29,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:29,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:29,321 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-10 02:54:29,321 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-10 02:54:29,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:29,323 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71316 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-10 02:54:29,323 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 161416 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-10 02:54:29,323 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/B is initiating minor compaction (all files) 2023-04-10 02:54:29,323 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/A is initiating minor compaction (all files) 2023-04-10 02:54:29,323 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/B in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:29,323 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/A in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:29,323 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/6df9067e043e49a5888b5a77b27fbb1e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/eaa106e1299a46d1966ab13df029cb28, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/058bda01b35741dfa2e0c3e22c71533b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/985ab9a5fb2e4bf886161c356e519108, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/bc6c1919b19943d98c7ead72f4618e70, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/c428dc41a1834d909a831b0c66b481a0] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=69.6 K 2023-04-10 02:54:29,323 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/2f1a2c282e2643a3af0b9a3b1d4e3902, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/c943a7bb010d490b947669e76977f91b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/e4952c6765614d629fc9be85dd4baf67, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/26e508a7ec2a4823abd248d565e5eaaa, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/51a235e84cef45449ae69d492fbdb4dd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/98d5ed57ccee439db0667428fe6c9665] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=157.6 K 2023-04-10 02:54:29,324 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6df9067e043e49a5888b5a77b27fbb1e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681095263337 2023-04-10 02:54:29,324 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 2f1a2c282e2643a3af0b9a3b1d4e3902, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681095263337 2023-04-10 02:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,324 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting eaa106e1299a46d1966ab13df029cb28, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1681095263358 2023-04-10 02:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:29,325 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting c943a7bb010d490b947669e76977f91b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1681095263358 2023-04-10 02:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@374d2639[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:29,325 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:29,325 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,325 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 058bda01b35741dfa2e0c3e22c71533b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681095264502 2023-04-10 02:54:29,326 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting e4952c6765614d629fc9be85dd4baf67, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681095264502 2023-04-10 02:54:29,326 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 985ab9a5fb2e4bf886161c356e519108, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681095265150 2023-04-10 02:54:29,326 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting bc6c1919b19943d98c7ead72f4618e70, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681095266277 2023-04-10 02:54:29,327 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,327 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:29,327 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:29,327 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:29,327 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 26e508a7ec2a4823abd248d565e5eaaa, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681095265150 2023-04-10 02:54:29,327 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting c428dc41a1834d909a831b0c66b481a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681095267415 2023-04-10 02:54:29,327 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,327 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:29,328 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:29,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:29,328 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 51a235e84cef45449ae69d492fbdb4dd, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681095266277 2023-04-10 02:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:29,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:29,329 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 98d5ed57ccee439db0667428fe6c9665, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681095267413 2023-04-10 02:54:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:29,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:29,334 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:29,334 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:29,334 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:29,334 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:29,334 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:29,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:29,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:29,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,337 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,339 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:29,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,342 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,342 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,342 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,342 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,342 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,342 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:29,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:29,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:29,342 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:29,342 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,342 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:29,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:29,342 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:29,343 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:29,343 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:29,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:29,352 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#B#compaction#144 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:29,373 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#A#compaction#145 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:29,443 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:29,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:29,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:29,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:29,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:29,446 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:29,446 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:29,446 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:29,446 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:29,447 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,447 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,448 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:29,448 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:29,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:29,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:29,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:29,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:29,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:29,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:29,451 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:29,451 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:29,451 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-10 02:54:29,451 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:29,451 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:29,451 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:29,451 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:29,451 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:29,452 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:29,466 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410a35b83d161bc4568b7af35181f2de4a5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a35b83d161bc4568b7af35181f2de4a5 2023-04-10 02:54:29,771 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/d73ea016e6ad4e26949e03c161dacc1b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d73ea016e6ad4e26949e03c161dacc1b 2023-04-10 02:54:29,776 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/B of a944754ff766fcb82f3c238c462e40cc into d73ea016e6ad4e26949e03c161dacc1b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:29,776 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:29,776 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/B, priority=10, startTime=1681095269321; duration=0sec 2023-04-10 02:54:29,776 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:29,776 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-10 02:54:29,778 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71316 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-10 02:54:29,778 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/C is initiating minor compaction (all files) 2023-04-10 02:54:29,778 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/C in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:29,778 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/259c48e4c3a649d1b9aa2da203082bee, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/5cdaa54fa5c94479a4095b5a2416e8c6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/20ba4388b40f46509623cc0a316ca48f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/3595ff077d144d0a9108495086d0cad0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/c5c96b26056b4130a418f78e2d4db46b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/d71a3d8b28ca45658a076a134c4f9d63] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=69.6 K 2023-04-10 02:54:29,779 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 259c48e4c3a649d1b9aa2da203082bee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681095263337 2023-04-10 02:54:29,779 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 5cdaa54fa5c94479a4095b5a2416e8c6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1681095263358 2023-04-10 02:54:29,779 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 20ba4388b40f46509623cc0a316ca48f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681095264502 2023-04-10 02:54:29,780 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 3595ff077d144d0a9108495086d0cad0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681095265150 2023-04-10 02:54:29,780 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting c5c96b26056b4130a418f78e2d4db46b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681095266277 2023-04-10 02:54:29,780 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d71a3d8b28ca45658a076a134c4f9d63, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681095267415 2023-04-10 02:54:29,795 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#C#compaction#147 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:29,804 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/b180dd1bf8474d4db026fc4c1e56bf0e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b180dd1bf8474d4db026fc4c1e56bf0e 2023-04-10 02:54:29,810 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/A of a944754ff766fcb82f3c238c462e40cc into b180dd1bf8474d4db026fc4c1e56bf0e(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:29,810 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:29,810 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/A, priority=10, startTime=1681095269321; duration=0sec 2023-04-10 02:54:29,810 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:29,871 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=115, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/73ada6a07dcb49f1a6b55ed7da880bc7 2023-04-10 02:54:30,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:30,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:30,208 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/882226a8ad454e18b5e2598817da510d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/882226a8ad454e18b5e2598817da510d 2023-04-10 02:54:30,213 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/C of a944754ff766fcb82f3c238c462e40cc into 882226a8ad454e18b5e2598817da510d(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:30,213 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:30,213 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/C, priority=10, startTime=1681095269321; duration=0sec 2023-04-10 02:54:30,213 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:30,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095330219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095330219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095330220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095330220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,282 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/1dcb489f700643d8bea75bae38a9ccb3 2023-04-10 02:54:30,300 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/7ec80126129845f0973c2423455f7a0e 2023-04-10 02:54:30,310 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/73ada6a07dcb49f1a6b55ed7da880bc7 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/73ada6a07dcb49f1a6b55ed7da880bc7 2023-04-10 02:54:30,317 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/73ada6a07dcb49f1a6b55ed7da880bc7, entries=150, sequenceid=115, filesize=25.1 K 2023-04-10 02:54:30,318 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/1dcb489f700643d8bea75bae38a9ccb3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/1dcb489f700643d8bea75bae38a9ccb3 2023-04-10 02:54:30,325 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/1dcb489f700643d8bea75bae38a9ccb3, entries=150, sequenceid=115, filesize=11.6 K 2023-04-10 02:54:30,326 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/7ec80126129845f0973c2423455f7a0e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/7ec80126129845f0973c2423455f7a0e 2023-04-10 02:54:30,332 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/7ec80126129845f0973c2423455f7a0e, entries=150, sequenceid=115, filesize=11.6 K 2023-04-10 02:54:30,333 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for a944754ff766fcb82f3c238c462e40cc in 882ms, sequenceid=115, compaction requested=false 2023-04-10 02:54:30,333 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:30,333 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:30,333 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:30,333 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:30,333 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,343 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:30,343 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:30,345 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,346 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,346 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:30,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cd26b5a[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:30,347 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:30,348 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,350 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,350 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:30,350 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:30,350 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:30,350 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,350 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:30,351 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:30,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,351 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:30,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,351 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:30,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:30,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:30,354 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:30,354 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:30,354 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:30,354 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:30,354 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:30,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:30,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:30,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:30,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,357 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,358 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:30,359 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-10 02:54:30,359 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:30,360 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:30,360 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:30,360 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:30,360 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:30,360 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:30,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,363 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:30,363 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:30,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:30,363 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,363 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,363 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,363 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,363 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,363 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,363 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:30,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:30,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:30,364 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:30,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:30,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:30,364 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:30,364 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:30,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095330367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095330368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095330368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095330370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095330434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,438 DEBUG [Thread-937] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:54:30,464 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:30,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:30,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:30,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:30,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:30,468 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:30,468 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:30,468 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:30,468 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:30,468 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,468 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,470 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:30,470 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:30,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:30,471 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,471 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:30,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:30,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:30,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:30,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:30,472 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:30,472 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:30,472 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:30,472 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:30,472 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:30,472 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:30,472 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:30,472 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:30,472 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:30,472 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:30,472 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:30,473 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,473 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:30,473 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:30,474 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,474 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,475 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,475 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:30,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f25f016[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:30,476 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:30,476 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095330474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,477 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,477 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095330474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,477 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:30,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095330474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,477 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:30,477 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:30,477 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:30,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095330475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:30,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:30,483 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:30,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:30,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:30,485 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:30,485 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:30,485 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:30,485 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:30,485 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:30,487 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,487 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,487 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:30,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:30,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:30,487 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,491 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,491 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:30,491 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,491 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,491 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,491 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,491 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:30,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:30,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:30,491 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:30,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:30,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:30,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:30,491 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:30,492 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:30,492 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:30,592 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:30,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:30,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:30,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:30,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:30,596 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:30,596 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:30,596 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:30,596 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:30,597 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,597 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,598 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:30,598 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:30,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:30,599 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,599 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:30,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:30,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:30,600 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:30,600 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:30,600 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:30,600 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:30,601 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:30,601 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:30,601 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:30,601 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:30,601 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:30,602 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:30,602 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:30,602 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,602 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:30,602 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:30,604 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,604 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,604 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,604 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2642b480[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:30,606 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:30,606 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,607 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,608 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:30,608 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:30,608 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:30,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:30,609 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,609 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:30,609 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:30,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,609 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:30,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:30,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:30,613 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:30,613 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:30,613 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:30,613 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:30,613 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:30,614 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,614 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,614 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:30,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:30,615 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,616 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,620 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:30,620 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:30,620 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,620 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:30,620 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:30,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:30,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:30,620 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,620 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,620 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,620 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,620 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:30,620 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:30,620 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:30,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:30,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:30,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095330678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095330678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095330679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095330680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,720 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:30,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:30,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:30,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:30,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:30,724 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:30,724 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:30,724 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:30,724 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:30,724 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,724 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,725 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:30,725 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:30,726 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,726 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:30,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:30,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:30,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:30,727 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:30,727 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:30,727 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:30,727 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:30,727 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:30,728 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:30,728 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:30,729 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:30,729 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:30,729 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:30,729 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:30,729 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,729 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:30,729 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:30,730 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,730 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,731 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:30,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:30,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7860329f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:30,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:30,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,734 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,734 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:30,734 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:30,734 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:30,734 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,734 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:30,736 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,736 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:30,736 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,736 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:30,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:30,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:30,739 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:30,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:30,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:30,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:30,739 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:30,741 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,741 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:30,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,741 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,741 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,743 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:30,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:30,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:30,746 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:30,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:30,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:30,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:30,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:30,747 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:30,747 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:30,793 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410577e1e32583146cf8593b1bf726cb28a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410577e1e32583146cf8593b1bf726cb28a 2023-04-10 02:54:30,847 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:30,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:30,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:30,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:30,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:30,850 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:30,850 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:30,850 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:30,850 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:30,851 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,851 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,853 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,853 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:30,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:30,853 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,853 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:30,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:30,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:30,855 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:30,855 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:30,855 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:30,855 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:30,855 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:30,855 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:30,855 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:30,855 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:30,855 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:30,855 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:30,855 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:30,855 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,855 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:30,855 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:30,857 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,857 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,857 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,857 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:30,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17871e72[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:30,859 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:30,859 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,860 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:30,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:30,861 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,868 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:30,868 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:30,868 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:30,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:30,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:30,872 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:30,872 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:30,872 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:30,872 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:30,872 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:30,877 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,877 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:30,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,877 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,879 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,881 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,883 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,883 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,883 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:30,883 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:30,883 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:30,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:30,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:30,883 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:30,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:30,884 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,884 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,884 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:30,884 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:30,884 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:30,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095330981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,984 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:30,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:30,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095330982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095330984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:30,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095330985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:30,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:30,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:30,988 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:30,988 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:30,988 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:30,988 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:30,989 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,989 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,993 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:30,993 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:30,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,994 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,994 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:30,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:30,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:30,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:30,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:30,995 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:30,995 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:30,995 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:30,995 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:30,997 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,997 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:30,998 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:30,998 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:30,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:30,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4772e9d4[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:30,999 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:30,999 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,000 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:31,000 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,001 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:31,001 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:31,002 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,001 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,002 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:31,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:31,005 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:31,005 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:31,005 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:31,005 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:31,005 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:31,013 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,013 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,013 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,014 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,015 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:31,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:31,017 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,018 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,018 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,018 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:31,018 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:31,018 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,018 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:31,118 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:31,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:31,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:31,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:31,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:31,121 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:31,121 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,121 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:31,121 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:31,122 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,122 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,123 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,123 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,123 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,123 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:31,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:31,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:31,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:31,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:31,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:31,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:31,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:31,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:31,125 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,125 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:31,125 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:31,125 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:31,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:31,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:31,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:31,127 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:31,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7427a30e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:31,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:31,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,131 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,131 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:31,131 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:31,131 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:31,131 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,131 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:31,132 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:31,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:31,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:31,135 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:31,136 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:31,136 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:31,136 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:31,136 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:31,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,138 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,139 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:31,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:31,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,141 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:31,142 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:31,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:31,141 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,199 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=139, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/b002b2e7673b4245bc0f5886ee2038c1 2023-04-10 02:54:31,242 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:31,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:31,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:31,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:31,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:31,245 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:31,245 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,245 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:31,245 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:31,246 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,246 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,248 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,249 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,249 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:31,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:31,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:31,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:31,251 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:31,251 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,251 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:31,257 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:31,257 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:31,258 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,258 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:31,258 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:31,258 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:31,260 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:31,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d2a93df[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:31,263 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:31,264 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,265 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,265 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:31,265 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,265 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:31,265 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:31,265 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:31,266 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:31,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,266 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,266 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:31,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:31,270 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:31,270 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:31,270 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:31,270 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:31,270 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:31,273 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,273 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,274 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,275 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,280 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,281 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,281 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,281 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,281 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,281 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,281 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,281 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,281 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,281 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,282 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:31,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:31,282 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:31,282 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:31,382 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:31,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:31,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:31,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:31,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:31,386 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:31,386 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,386 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:31,386 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:31,387 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,387 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,388 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,388 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,389 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,389 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:31,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:31,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:31,391 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:31,391 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,391 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:31,393 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,393 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:31,393 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:31,393 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:31,395 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,395 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,396 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,396 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:31,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f17886[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:31,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:31,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,398 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:31,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:31,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:31,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:31,400 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:31,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:31,403 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:31,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:31,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:31,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:31,403 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:31,406 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,407 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,408 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:31,408 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,408 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:31,408 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,408 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,409 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,412 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,412 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,412 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:31,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:31,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:31,412 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,413 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:31,413 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:31,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:31,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095331484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:31,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095331487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:31,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095331487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:31,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095331490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,513 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:31,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:31,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:31,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:31,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:31,517 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:31,517 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,517 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:31,517 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:31,517 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,517 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,519 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,519 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,519 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,519 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:31,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:31,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:31,521 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,521 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:31,521 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:31,521 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,526 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:31,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@729d0945[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:31,529 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:31,529 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,530 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,531 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:31,531 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,531 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:31,531 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:31,531 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:31,532 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,532 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:31,532 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,532 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:31,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:31,535 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:31,535 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:31,535 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:31,535 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:31,535 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:31,537 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,537 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,537 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,538 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,541 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:31,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:31,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:31,547 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:31,547 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:31,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,613 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/87cec3859eb74747a6b6bef9e8694f39 2023-04-10 02:54:31,647 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:31,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:31,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:31,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:31,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:31,650 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:31,650 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,650 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:31,650 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:31,650 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,651 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,652 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,652 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,652 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,652 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:31,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:31,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:31,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:31,653 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:31,653 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,653 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:31,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:31,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:31,654 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,654 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:31,654 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:31,654 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:31,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:31,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:31,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:31,655 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,655 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,656 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,656 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:31,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bebebcb[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:31,657 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:31,657 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,659 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,659 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:31,659 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:31,659 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,659 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:31,659 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:31,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,661 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:31,661 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,661 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:31,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:31,663 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:31,663 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:31,663 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:31,663 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:31,663 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:31,665 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,665 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,665 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,666 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,669 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,678 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:31,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:31,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:31,679 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:31,679 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:31,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,779 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:31,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:31,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:31,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:31,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:31,783 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:31,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:31,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:31,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,785 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,785 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,785 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,785 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:31,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:31,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:31,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:31,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:31,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:31,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:31,787 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,787 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:31,787 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:31,787 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,787 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:31,787 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:31,787 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,787 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:31,787 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:31,788 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,789 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,789 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:31,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:31,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62863302[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:31,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:31,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,793 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,793 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:31,793 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:31,793 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:31,793 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,793 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:31,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:31,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:31,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:31,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:31,797 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:31,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:31,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:31,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:31,797 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:31,798 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,798 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,798 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,802 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,802 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,802 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,802 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,802 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,802 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,802 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:31,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:31,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:31,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,802 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,802 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,803 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:31,803 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:31,803 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,903 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:31,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:31,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:31,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:31,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:31,906 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:31,906 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,906 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:31,906 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:31,906 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,906 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,908 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,908 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,909 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,909 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:31,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:31,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:31,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:31,910 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,910 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:31,910 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:31,910 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,911 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:31,911 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:31,912 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,912 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,912 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,912 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:31,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:31,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1a529c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:31,914 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:31,914 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,915 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:31,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:31,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:31,915 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,915 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:31,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,916 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:31,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,916 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,916 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:31,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:31,919 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:31,919 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:31,919 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:31,919 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:31,919 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:31,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:31,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,922 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:31,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,925 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:31,925 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:31,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:31,925 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:31,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:31,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:31,925 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:31,925 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:31,926 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:31,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:31,926 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:31,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:31,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:31,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:32,025 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/19a33b25e5694f87ba6eb98d60c8cb0c 2023-04-10 02:54:32,026 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:32,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:32,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:32,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:32,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:32,029 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:32,030 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:32,030 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:32,030 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:32,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/b002b2e7673b4245bc0f5886ee2038c1 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b002b2e7673b4245bc0f5886ee2038c1 2023-04-10 02:54:32,031 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:32,031 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:32,032 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:32,032 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:32,033 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,033 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:32,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:32,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:32,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:32,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:32,034 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:32,034 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:32,034 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:32,034 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:32,034 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:32,034 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:32,034 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:32,035 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:32,035 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:32,035 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:32,035 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:32,035 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,035 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:32,035 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:32,035 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b002b2e7673b4245bc0f5886ee2038c1, entries=200, sequenceid=139, filesize=32.1 K 2023-04-10 02:54:32,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/87cec3859eb74747a6b6bef9e8694f39 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/87cec3859eb74747a6b6bef9e8694f39 2023-04-10 02:54:32,037 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:32,037 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:32,038 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:32,038 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:32,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:32,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@664b9a58[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:32,041 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:32,041 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:32,043 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:32,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:32,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:32,043 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,043 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:32,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:32,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:32,045 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,045 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:32,045 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:32,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/87cec3859eb74747a6b6bef9e8694f39, entries=150, sequenceid=139, filesize=11.8 K 2023-04-10 02:54:32,045 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:32,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:32,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:32,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:32,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:32,048 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:32,048 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:32,048 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:32,049 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:32,049 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:32,049 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/19a33b25e5694f87ba6eb98d60c8cb0c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/19a33b25e5694f87ba6eb98d60c8cb0c 2023-04-10 02:54:32,050 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:32,050 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:32,050 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:32,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:32,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:32,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:32,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:32,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:32,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:32,051 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:32,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:32,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,052 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,055 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:32,055 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:32,055 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:32,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:32,055 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:32,055 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:32,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:32,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:32,055 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:32,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:32,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:32,055 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,055 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:32,055 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:32,055 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:32,055 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:32,055 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:32,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:32,056 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/19a33b25e5694f87ba6eb98d60c8cb0c, entries=150, sequenceid=139, filesize=11.8 K 2023-04-10 02:54:32,057 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for a944754ff766fcb82f3c238c462e40cc in 1698ms, sequenceid=139, compaction requested=true 2023-04-10 02:54:32,057 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:32,057 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:32,057 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:32,057 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:32,057 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:32,057 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:32,058 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84584 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:32,058 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/A is initiating minor compaction (all files) 2023-04-10 02:54:32,059 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/A in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:32,059 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b180dd1bf8474d4db026fc4c1e56bf0e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/73ada6a07dcb49f1a6b55ed7da880bc7, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b002b2e7673b4245bc0f5886ee2038c1] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=82.6 K 2023-04-10 02:54:32,059 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:32,059 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/B is initiating minor compaction (all files) 2023-04-10 02:54:32,059 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/B in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:32,059 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d73ea016e6ad4e26949e03c161dacc1b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/1dcb489f700643d8bea75bae38a9ccb3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/87cec3859eb74747a6b6bef9e8694f39] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=35.2 K 2023-04-10 02:54:32,059 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting b180dd1bf8474d4db026fc4c1e56bf0e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681095267415 2023-04-10 02:54:32,060 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d73ea016e6ad4e26949e03c161dacc1b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681095267415 2023-04-10 02:54:32,060 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 73ada6a07dcb49f1a6b55ed7da880bc7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681095268040 2023-04-10 02:54:32,060 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 1dcb489f700643d8bea75bae38a9ccb3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681095268040 2023-04-10 02:54:32,060 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting b002b2e7673b4245bc0f5886ee2038c1, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681095270218 2023-04-10 02:54:32,061 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 87cec3859eb74747a6b6bef9e8694f39, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681095270218 2023-04-10 02:54:32,068 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#B#compaction#154 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:32,072 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#A#compaction#153 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:32,085 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/5e9b20c088af49459195955a55230219 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5e9b20c088af49459195955a55230219 2023-04-10 02:54:32,092 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/B of a944754ff766fcb82f3c238c462e40cc into 5e9b20c088af49459195955a55230219(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:32,092 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:32,092 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/B, priority=13, startTime=1681095272057; duration=0sec 2023-04-10 02:54:32,092 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:32,092 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:32,094 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:32,095 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/C is initiating minor compaction (all files) 2023-04-10 02:54:32,095 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/C in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:32,095 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/882226a8ad454e18b5e2598817da510d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/7ec80126129845f0973c2423455f7a0e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/19a33b25e5694f87ba6eb98d60c8cb0c] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=35.2 K 2023-04-10 02:54:32,095 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 882226a8ad454e18b5e2598817da510d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681095267415 2023-04-10 02:54:32,096 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 7ec80126129845f0973c2423455f7a0e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681095268040 2023-04-10 02:54:32,096 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 19a33b25e5694f87ba6eb98d60c8cb0c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681095270218 2023-04-10 02:54:32,108 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#C#compaction#155 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:32,108 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/ee6db7ce430b4a738df078ac0cc3e647 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/ee6db7ce430b4a738df078ac0cc3e647 2023-04-10 02:54:32,119 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/A of a944754ff766fcb82f3c238c462e40cc into ee6db7ce430b4a738df078ac0cc3e647(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:32,119 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:32,119 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/A, priority=13, startTime=1681095272057; duration=0sec 2023-04-10 02:54:32,119 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:32,121 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/a0fbd28555d9433ab4e9d576076e439f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/a0fbd28555d9433ab4e9d576076e439f 2023-04-10 02:54:32,130 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/C of a944754ff766fcb82f3c238c462e40cc into a0fbd28555d9433ab4e9d576076e439f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:32,130 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:32,130 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/C, priority=13, startTime=1681095272057; duration=0sec 2023-04-10 02:54:32,130 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:32,155 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:32,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:32,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:32,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:32,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:32,160 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:32,160 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:32,160 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:32,160 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:32,160 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:32,160 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:32,161 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,161 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:32,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:32,162 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,162 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:32,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:32,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:32,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:32,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:32,163 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:32,163 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:32,164 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:54:32,164 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:32,164 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:32,164 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:32,164 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:32,164 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:32,164 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:32,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:32,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:32,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:32,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095332502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681095272495, max timestamp=1681095272505Num uniques 859; 2023-04-10 02:54:32,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-10 02:54:32,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681095272495, max timestamp=1681095272505Num uniques 859; 2023-04-10 02:54:32,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-10 02:54:32,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681095272495, max timestamp=1681095272505Num uniques 859; 2023-04-10 02:54:32,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-10 02:54:32,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:32,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095332520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:32,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:32,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095332521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095332521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,580 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041095a58f20ab9d40ddb4d65ca3d7aa0eaf to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041095a58f20ab9d40ddb4d65ca3d7aa0eaf 2023-04-10 02:54:32,593 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=153, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/b2a9ba1195944567ad0563b712e064eb 2023-04-10 02:54:32,604 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/8f6ff9a8bf5a42e1b976e0a2688c0aa2 2023-04-10 02:54:32,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:32,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095332608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:32,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095332623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:32,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095332626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:32,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095332626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:32,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095332812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:32,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095332826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:32,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095332831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:32,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:32,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095332831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,015 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/68e6d63f18c346bf9bc6610360332c2f 2023-04-10 02:54:33,020 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/b2a9ba1195944567ad0563b712e064eb as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b2a9ba1195944567ad0563b712e064eb 2023-04-10 02:54:33,025 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b2a9ba1195944567ad0563b712e064eb, entries=150, sequenceid=153, filesize=25.3 K 2023-04-10 02:54:33,026 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/8f6ff9a8bf5a42e1b976e0a2688c0aa2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/8f6ff9a8bf5a42e1b976e0a2688c0aa2 2023-04-10 02:54:33,032 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/8f6ff9a8bf5a42e1b976e0a2688c0aa2, entries=150, sequenceid=153, filesize=11.8 K 2023-04-10 02:54:33,033 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/68e6d63f18c346bf9bc6610360332c2f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/68e6d63f18c346bf9bc6610360332c2f 2023-04-10 02:54:33,040 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/68e6d63f18c346bf9bc6610360332c2f, entries=150, sequenceid=153, filesize=11.8 K 2023-04-10 02:54:33,041 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for a944754ff766fcb82f3c238c462e40cc in 877ms, sequenceid=153, compaction requested=false 2023-04-10 02:54:33,042 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:33,042 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,042 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:33,042 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:33,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:33,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:33,047 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:33,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f5caabf[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:33,049 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:33,049 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,050 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:33,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:33,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:33,051 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,051 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:33,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:33,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,053 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:33,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:33,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:33,055 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:33,055 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:33,055 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:33,055 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:33,055 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:33,058 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,058 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,059 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,061 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,067 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:33,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:33,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,067 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:33,067 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:33,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:33,117 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=140.89 KB heapSize=386.16 KB 2023-04-10 02:54:33,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:33,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:33,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:33,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:33,118 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:33,118 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:33,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:33,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095333133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:33,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095333134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:33,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095333134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:33,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095333135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,167 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:33,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:33,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:33,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:33,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:33,171 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:33,171 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,171 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:33,171 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:33,172 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,172 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,173 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,173 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,174 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:33,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:33,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:33,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:33,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:33,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:33,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:33,175 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,175 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:33,175 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:33,175 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:33,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:33,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:33,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:33,177 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,177 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,177 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,177 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:33,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@130bfbba[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:33,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:33,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,180 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:33,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:33,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:33,180 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,180 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:33,182 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,182 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:33,182 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,182 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:33,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:33,184 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:33,185 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:33,185 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:33,185 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:33,185 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:33,186 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,186 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,186 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,187 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,188 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,190 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:33,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:33,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,191 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,191 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,191 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,191 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:33,191 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,191 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:33,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,191 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:33,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095333240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:33,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095333240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,291 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:33,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:33,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:33,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:33,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:33,295 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:33,295 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,295 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:33,295 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:33,295 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,295 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,298 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,298 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,298 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,298 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:33,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:33,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:33,299 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:33,299 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,299 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:33,299 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:33,299 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:33,299 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,299 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:33,299 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:33,300 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,300 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:33,300 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:33,300 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,300 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:33,300 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:33,301 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,301 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,301 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,301 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:33,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@138318eb[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:33,303 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:33,303 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,304 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,304 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:33,304 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:33,304 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:33,304 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,304 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:33,305 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,305 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,305 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,306 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:33,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:33,308 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:33,308 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:33,308 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:33,308 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:33,308 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:33,312 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,312 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,312 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,313 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,316 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,316 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,316 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,316 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,316 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,316 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,316 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:33,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:33,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,316 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,317 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,317 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,317 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:33,317 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:33,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,417 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:33,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:33,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:33,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:33,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:33,422 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:33,422 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,422 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:33,422 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:33,423 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,423 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,424 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,424 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:33,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:33,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:33,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:33,426 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:33,426 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,426 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:33,426 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:33,426 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:33,426 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,426 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:33,427 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:33,427 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,427 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:33,427 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:33,427 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,427 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:33,427 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:33,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,428 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,428 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,428 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:33,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2920d999[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:33,433 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:33,433 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,438 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,438 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:33,438 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:33,438 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:33,440 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:33,440 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,440 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,440 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,440 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,441 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:33,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:33,444 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:33,444 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:33,444 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:33,444 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:33,444 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:33,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:33,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095333443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:33,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095333444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,446 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,446 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,446 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,447 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,453 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,453 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,453 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,453 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,453 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:33,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:33,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,453 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,453 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:33,453 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:33,453 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,454 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,454 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,454 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:33,535 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041033dfc16ffb124aa3a4efae37be7990b4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041033dfc16ffb124aa3a4efae37be7990b4 2023-04-10 02:54:33,554 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:33,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:33,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:33,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:33,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:33,557 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:33,557 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,557 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:33,557 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:33,558 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,558 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,559 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,559 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,560 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,560 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:33,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:33,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:33,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:33,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:33,561 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:33,561 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,561 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:33,561 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:33,561 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:33,561 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,562 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:33,562 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:33,562 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,562 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:33,562 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:33,562 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,562 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:33,562 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:33,564 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,564 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,564 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,564 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33338750[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:33,567 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:33,567 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,568 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,568 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:33,568 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:33,568 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:33,568 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,568 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:33,571 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,571 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:33,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,571 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,571 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:33,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:33,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:33,576 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:33,576 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:33,576 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:33,576 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:33,576 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:33,577 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,577 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,578 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,578 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,580 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,583 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,583 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,583 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,584 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:33,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:33,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,584 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:33,584 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:33,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:33,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:33,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095333640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:33,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095333640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,684 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:33,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:33,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:33,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:33,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:33,687 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:33,687 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,687 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:33,687 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:33,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,690 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,690 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,690 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,690 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:33,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:33,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:33,691 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:33,691 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,691 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:33,691 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:33,691 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:33,691 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,691 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:33,691 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:33,691 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:33,694 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,694 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:33,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f297ca7[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:33,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:33,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:33,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:33,697 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,697 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:33,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:33,699 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:33,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:33,701 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:33,701 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:33,701 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:33,701 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:33,701 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:33,703 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,703 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,703 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,703 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,708 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,708 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,708 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:33,708 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,708 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,708 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:33,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:33,708 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,708 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,708 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,709 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,709 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:33,709 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:33,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:33,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095333745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:33,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095333747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,809 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:33,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:33,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:33,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:33,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:33,812 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:33,812 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,812 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:33,812 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:33,812 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,812 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,814 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,814 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,814 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:33,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:33,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:33,815 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:33,815 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,815 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:33,815 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:33,815 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:33,815 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,816 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:33,816 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:33,816 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,816 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:33,816 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:33,816 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,816 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:33,816 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:33,817 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,817 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@746667e9[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:33,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:33,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,820 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,820 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:33,820 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:33,820 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:33,820 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,820 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:33,823 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:33,823 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,823 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,823 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:33,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:33,825 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:33,826 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:33,826 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:33,826 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:33,826 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:33,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,830 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,832 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,834 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,834 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,834 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,834 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,834 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:33,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:33,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,834 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,834 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,835 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:33,835 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,835 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,835 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,835 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:33,935 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:33,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:33,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:33,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:33,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:33,938 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:33,938 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,938 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:33,938 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:33,939 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,939 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,940 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,940 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,940 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,941 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:33,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:33,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:33,942 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,942 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:33,942 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:33,942 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:33,944 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=187, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/34802c322c824c9b8890ccd396140d09 2023-04-10 02:54:33,945 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,945 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:33,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ba584ed[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:33,947 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:33,947 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:33,948 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,948 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:33,950 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:33,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:33,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:33,953 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:33,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:33,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:33,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:33,953 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:33,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:33,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:33,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,955 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:33,957 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/d41c7ef0236f4b218f5e210e336d7214 2023-04-10 02:54:33,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:33,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:33,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:33,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:33,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:33,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:33,960 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:33,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:33,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:33,961 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:33,961 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:33,961 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,961 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:33,961 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:33,976 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/21519a1cb100437380bbc32a707ca2a2 2023-04-10 02:54:33,980 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/34802c322c824c9b8890ccd396140d09 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/34802c322c824c9b8890ccd396140d09 2023-04-10 02:54:33,984 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/34802c322c824c9b8890ccd396140d09, entries=200, sequenceid=187, filesize=32.1 K 2023-04-10 02:54:33,985 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/d41c7ef0236f4b218f5e210e336d7214 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d41c7ef0236f4b218f5e210e336d7214 2023-04-10 02:54:33,989 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d41c7ef0236f4b218f5e210e336d7214, entries=150, sequenceid=187, filesize=11.8 K 2023-04-10 02:54:33,990 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/21519a1cb100437380bbc32a707ca2a2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/21519a1cb100437380bbc32a707ca2a2 2023-04-10 02:54:34,002 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/21519a1cb100437380bbc32a707ca2a2, entries=150, sequenceid=187, filesize=11.8 K 2023-04-10 02:54:34,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~386.11 KB/395376, currentSize=46.96 KB/48090 for a944754ff766fcb82f3c238c462e40cc in 886ms, sequenceid=187, compaction requested=true 2023-04-10 02:54:34,004 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:34,004 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:34,004 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:34,004 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:34,004 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:34,004 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:34,006 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84986 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:34,006 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/A is initiating minor compaction (all files) 2023-04-10 02:54:34,006 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:34,006 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/A in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:34,006 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/B is initiating minor compaction (all files) 2023-04-10 02:54:34,006 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/ee6db7ce430b4a738df078ac0cc3e647, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b2a9ba1195944567ad0563b712e064eb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/34802c322c824c9b8890ccd396140d09] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=83.0 K 2023-04-10 02:54:34,006 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/B in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:34,007 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5e9b20c088af49459195955a55230219, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/8f6ff9a8bf5a42e1b976e0a2688c0aa2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d41c7ef0236f4b218f5e210e336d7214] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=35.6 K 2023-04-10 02:54:34,008 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ee6db7ce430b4a738df078ac0cc3e647, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681095270218 2023-04-10 02:54:34,008 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 5e9b20c088af49459195955a55230219, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681095270218 2023-04-10 02:54:34,008 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting b2a9ba1195944567ad0563b712e064eb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681095270365 2023-04-10 02:54:34,008 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 8f6ff9a8bf5a42e1b976e0a2688c0aa2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681095270365 2023-04-10 02:54:34,008 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 34802c322c824c9b8890ccd396140d09, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681095272519 2023-04-10 02:54:34,009 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d41c7ef0236f4b218f5e210e336d7214, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681095272519 2023-04-10 02:54:34,027 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#B#compaction#163 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:34,031 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#A#compaction#162 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:34,045 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/115c5078929646e9b021db28cfbd47e8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/115c5078929646e9b021db28cfbd47e8 2023-04-10 02:54:34,051 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/B of a944754ff766fcb82f3c238c462e40cc into 115c5078929646e9b021db28cfbd47e8(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:34,051 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:34,052 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/B, priority=13, startTime=1681095274004; duration=0sec 2023-04-10 02:54:34,052 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:34,052 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:34,053 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:34,053 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/C is initiating minor compaction (all files) 2023-04-10 02:54:34,054 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/C in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:34,054 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/a0fbd28555d9433ab4e9d576076e439f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/68e6d63f18c346bf9bc6610360332c2f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/21519a1cb100437380bbc32a707ca2a2] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=35.6 K 2023-04-10 02:54:34,054 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting a0fbd28555d9433ab4e9d576076e439f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681095270218 2023-04-10 02:54:34,054 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 68e6d63f18c346bf9bc6610360332c2f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681095270365 2023-04-10 02:54:34,055 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 21519a1cb100437380bbc32a707ca2a2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681095272519 2023-04-10 02:54:34,061 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:34,062 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#C#compaction#164 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:34,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:34,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:34,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:34,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:34,068 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:34,068 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:34,068 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:34,068 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:34,069 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:34,069 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:34,070 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:34,070 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:34,071 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,071 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:34,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:34,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:34,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:34,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:34,080 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:34,080 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:34,080 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:34,080 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:34,080 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:34,080 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:34,080 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:34,081 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:54:34,081 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:34,081 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:34,081 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:34,081 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:34,082 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:34,082 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:34,090 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/bcdfd786491f44bfadaa2f211a79b67d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/bcdfd786491f44bfadaa2f211a79b67d 2023-04-10 02:54:34,095 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/C of a944754ff766fcb82f3c238c462e40cc into bcdfd786491f44bfadaa2f211a79b67d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:34,095 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:34,095 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/C, priority=13, startTime=1681095274004; duration=0sec 2023-04-10 02:54:34,095 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:34,098 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410d2274c0ed5284c50a9883670aafbdfc9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410d2274c0ed5284c50a9883670aafbdfc9 2023-04-10 02:54:34,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:34,258 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:34,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095274274, max timestamp=1681095274288Num uniques 859; 2023-04-10 02:54:34,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095274274, max timestamp=1681095274288Num uniques 859; 2023-04-10 02:54:34,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:54:34,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:54:34,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095274274, max timestamp=1681095274288Num uniques 859; 2023-04-10 02:54:34,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:54:34,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:34,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53892 deadline: 1681095334331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:34,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095334331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:34,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095334432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:34,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095334432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:34,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53914 deadline: 1681095334464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,469 DEBUG [Thread-937] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:54:34,515 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=198, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/a44cccd818c94493af87e52eb556b234 2023-04-10 02:54:34,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:34,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095334635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:34,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53894 deadline: 1681095334636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:34,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095334647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:34,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095334650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,894 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/da0c282a0c52472f94357fe89467e9cc as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/da0c282a0c52472f94357fe89467e9cc 2023-04-10 02:54:34,900 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/A of a944754ff766fcb82f3c238c462e40cc into da0c282a0c52472f94357fe89467e9cc(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:34,900 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:34,900 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/A, priority=13, startTime=1681095274004; duration=0sec 2023-04-10 02:54:34,900 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:34,926 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/5e9fce30affb45dab1ff7627bc2cc85b 2023-04-10 02:54:34,942 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/38edeac5f2dd41d09cefd9caf08b15f5 2023-04-10 02:54:34,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:34,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095334940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:34,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095334941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,947 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/a44cccd818c94493af87e52eb556b234 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/a44cccd818c94493af87e52eb556b234 2023-04-10 02:54:34,951 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/a44cccd818c94493af87e52eb556b234, entries=100, sequenceid=198, filesize=18.5 K 2023-04-10 02:54:34,953 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/5e9fce30affb45dab1ff7627bc2cc85b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5e9fce30affb45dab1ff7627bc2cc85b 2023-04-10 02:54:34,958 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5e9fce30affb45dab1ff7627bc2cc85b, entries=100, sequenceid=198, filesize=9.4 K 2023-04-10 02:54:34,959 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/38edeac5f2dd41d09cefd9caf08b15f5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/38edeac5f2dd41d09cefd9caf08b15f5 2023-04-10 02:54:34,964 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/38edeac5f2dd41d09cefd9caf08b15f5, entries=100, sequenceid=198, filesize=9.4 K 2023-04-10 02:54:34,965 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a944754ff766fcb82f3c238c462e40cc in 884ms, sequenceid=198, compaction requested=false 2023-04-10 02:54:34,966 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:34,966 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:34,971 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:34,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:34,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@740391ce[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:34,974 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:34,974 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:34,975 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:34,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:34,975 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:34,976 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:34,976 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:34,976 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,976 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:34,978 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:34,978 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:34,979 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:34,979 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:34,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:34,981 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:34,981 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:34,981 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:34,981 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:34,982 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:34,984 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:34,984 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:34,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:34,984 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:34,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:34,987 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,989 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:34,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:34,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:34,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:34,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:34,992 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:34,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:34,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:34,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:34,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:34,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:34,992 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:34,992 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:34,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:34,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:35,093 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:35,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:35,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:35,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:35,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:35,096 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:35,096 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:35,096 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:35,096 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:35,096 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,097 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:35,098 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,098 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:35,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:35,098 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,098 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:35,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:35,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:35,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:35,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:35,099 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:35,099 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:35,099 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:35,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:35,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:35,100 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:35,100 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:35,100 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-10 02:54:35,100 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:35,100 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:35,100 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:35,100 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:35,100 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:35,100 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:35,122 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410107dfd41acb24797b23daf99c31817dd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410107dfd41acb24797b23daf99c31817dd 2023-04-10 02:54:35,127 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=235, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/9e516e3a294b46cc8fbb9c92f733406f 2023-04-10 02:54:35,136 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/5aca45fb63ab47c3b6b56656b3001417 2023-04-10 02:54:35,146 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/6be3456d2b7542d29253bb8e96f877c7 2023-04-10 02:54:35,151 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/9e516e3a294b46cc8fbb9c92f733406f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/9e516e3a294b46cc8fbb9c92f733406f 2023-04-10 02:54:35,158 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/9e516e3a294b46cc8fbb9c92f733406f, entries=150, sequenceid=235, filesize=25.3 K 2023-04-10 02:54:35,159 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/5aca45fb63ab47c3b6b56656b3001417 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5aca45fb63ab47c3b6b56656b3001417 2023-04-10 02:54:35,163 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5aca45fb63ab47c3b6b56656b3001417, entries=150, sequenceid=235, filesize=11.8 K 2023-04-10 02:54:35,164 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/6be3456d2b7542d29253bb8e96f877c7 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/6be3456d2b7542d29253bb8e96f877c7 2023-04-10 02:54:35,184 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/6be3456d2b7542d29253bb8e96f877c7, entries=150, sequenceid=235, filesize=11.8 K 2023-04-10 02:54:35,185 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=0 B/0 for a944754ff766fcb82f3c238c462e40cc in 85ms, sequenceid=235, compaction requested=true 2023-04-10 02:54:35,185 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:35,186 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:35,186 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:35,186 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:35,186 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,186 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:35,186 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:35,189 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,189 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:35,189 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,189 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:35,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:35,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@407b17ad[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:35,191 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:35,191 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,192 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:35,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:35,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:35,193 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,193 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:35,194 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:35,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:35,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:35,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:35,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:35,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:35,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:35,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:35,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:35,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:35,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:35,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:35,197 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:35,197 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:35,197 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:35,197 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:35,197 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:35,199 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,199 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:35,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,199 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:35,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:35,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:35,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:35,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:35,201 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:35,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:35,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,202 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:35,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:35,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:35,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:35,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:35,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:35,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:35,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:35,210 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:35,211 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:35,211 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:35,211 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:35,311 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:35,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:35,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:35,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:35,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:35,316 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:35,316 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:35,317 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:35,317 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:35,317 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,317 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:35,318 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,318 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:35,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:35,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:35,319 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,319 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:35,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:35,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:35,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:35,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:35,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:35,320 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:35,320 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:35,320 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:35,321 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:35,321 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:35,321 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:35,321 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:35,321 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:35,321 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:35,321 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:35,321 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:35,321 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,321 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:35,321 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,323 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:35,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:35,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e57964[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:35,325 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:35,325 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,328 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:35,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:35,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:35,328 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,328 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:35,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:35,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:35,330 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:35,329 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:35,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:35,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:35,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:35,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:35,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:35,332 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:35,332 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:35,333 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:35,333 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:35,333 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:35,334 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,334 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:35,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:35,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,334 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:35,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:35,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:35,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:35,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,336 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:35,337 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:35,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:35,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:35,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:35,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,340 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:35,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:35,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:35,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:35,341 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:35,341 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:35,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:35,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:35,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,341 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:35,341 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:35,441 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:35,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:35,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:35,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:35,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:35,445 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:35,445 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:35,445 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:35,445 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:35,445 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,446 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:35,448 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:35,448 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:35,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:35,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:35,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:35,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:35,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:35,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:35,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:35,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:35,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:35,450 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:35,450 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:35,450 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:35,450 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:35,450 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:35,450 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:35,450 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:35,451 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-10 02:54:35,452 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:35,452 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:35,452 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:35,452 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:35,452 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:35,452 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:35,478 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410a568ad8f84864746b2a1fb1ba12fdd79 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a568ad8f84864746b2a1fb1ba12fdd79 2023-04-10 02:54:35,488 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/2124b1fd6cac4205a7687a0f14b0a81a 2023-04-10 02:54:35,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:35,494 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:35,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681095275452, max timestamp=1681095275519Num uniques 859; 2023-04-10 02:54:35,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-10 02:54:35,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681095275452, max timestamp=1681095275519Num uniques 859; 2023-04-10 02:54:35,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681095275452, max timestamp=1681095275519Num uniques 859; 2023-04-10 02:54:35,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-10 02:54:35,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-10 02:54:35,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:35,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095335541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:35,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095335542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:35,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095335644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:35,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095335644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:35,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095335848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:35,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095335848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:35,900 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/d056f82b3e7143bf9e8b4e77dace8fcf 2023-04-10 02:54:36,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:36,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095336151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:36,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095336151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,310 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/55f7f1cc13b44a7b824acc35d4680296 2023-04-10 02:54:36,315 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/2124b1fd6cac4205a7687a0f14b0a81a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/2124b1fd6cac4205a7687a0f14b0a81a 2023-04-10 02:54:36,319 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/2124b1fd6cac4205a7687a0f14b0a81a, entries=50, sequenceid=240, filesize=11.6 K 2023-04-10 02:54:36,320 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/d056f82b3e7143bf9e8b4e77dace8fcf as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d056f82b3e7143bf9e8b4e77dace8fcf 2023-04-10 02:54:36,325 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d056f82b3e7143bf9e8b4e77dace8fcf, entries=50, sequenceid=240, filesize=7.1 K 2023-04-10 02:54:36,327 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/55f7f1cc13b44a7b824acc35d4680296 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/55f7f1cc13b44a7b824acc35d4680296 2023-04-10 02:54:36,332 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/55f7f1cc13b44a7b824acc35d4680296, entries=50, sequenceid=240, filesize=7.1 K 2023-04-10 02:54:36,333 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=194.56 KB/199230 for a944754ff766fcb82f3c238c462e40cc in 883ms, sequenceid=240, compaction requested=true 2023-04-10 02:54:36,333 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:36,333 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:36,333 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:36,333 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:36,333 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,333 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:36,333 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:36,336 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:36,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:36,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:36,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70901799[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:36,339 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:36,339 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:36,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:36,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:36,340 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:36,340 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:36,340 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:36,341 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,341 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:36,342 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,342 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:36,342 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:36,342 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:36,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:36,345 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:36,345 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:36,345 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:36,345 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:36,345 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:36,347 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:36,347 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:36,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:36,347 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:36,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:36,348 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:36,350 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,353 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,353 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:36,353 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:36,353 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:36,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:36,353 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:36,353 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:36,353 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:36,353 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:36,353 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:36,353 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:36,353 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:36,353 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:36,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:36,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:36,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:36,454 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:36,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:36,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:36,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:36,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:36,457 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:36,457 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:36,457 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:36,457 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:36,457 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:36,458 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:36,459 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:36,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:36,459 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,459 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,459 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:36,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:36,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:36,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:36,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:36,461 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:36,461 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:36,461 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:36,461 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:36,461 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:36,461 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:36,461 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:36,461 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=194.56 KB heapSize=498.66 KB 2023-04-10 02:54:36,461 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:36,461 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:36,462 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:36,462 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:36,462 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:36,462 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:36,485 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041052c19ca38d1d4a37b878b37c69153216 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041052c19ca38d1d4a37b878b37c69153216 2023-04-10 02:54:36,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:36,654 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:36,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:36,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095336655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:36,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095336655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:36,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095336663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,667 DEBUG [Thread-945] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:54:36,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:36,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095336668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,672 DEBUG [Thread-941] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:54:36,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:36,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095336757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:36,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095336757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,906 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=272, memsize=64.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/05bfb749a70240569a37492d5e60d008 2023-04-10 02:54:36,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:36,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095336962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:36,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:36,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095336962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:37,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095337269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:37,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095337269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,339 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/025003e13fe24b22a946b475dcfb4d23 2023-04-10 02:54:37,752 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/57eb9cfbe2104822ab24995b451afb90 2023-04-10 02:54:37,757 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/05bfb749a70240569a37492d5e60d008 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/05bfb749a70240569a37492d5e60d008 2023-04-10 02:54:37,761 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/05bfb749a70240569a37492d5e60d008, entries=150, sequenceid=272, filesize=25.4 K 2023-04-10 02:54:37,761 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/025003e13fe24b22a946b475dcfb4d23 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/025003e13fe24b22a946b475dcfb4d23 2023-04-10 02:54:37,765 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/025003e13fe24b22a946b475dcfb4d23, entries=150, sequenceid=272, filesize=11.9 K 2023-04-10 02:54:37,766 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/57eb9cfbe2104822ab24995b451afb90 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/57eb9cfbe2104822ab24995b451afb90 2023-04-10 02:54:37,771 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/57eb9cfbe2104822ab24995b451afb90, entries=150, sequenceid=272, filesize=11.9 K 2023-04-10 02:54:37,772 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~194.56 KB/199230, heapSize ~498.61 KB/510576, currentSize=13.42 KB/13740 for a944754ff766fcb82f3c238c462e40cc in 1311ms, sequenceid=272, compaction requested=true 2023-04-10 02:54:37,772 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:37,772 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:37,774 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:37,774 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,774 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:37,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:37,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:37,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:37,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@187565a4[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:37,776 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:37,776 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,777 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,777 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:37,777 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:37,777 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:37,777 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,777 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:37,780 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:37,780 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:37,780 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,780 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:37,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:37,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:37,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:37,783 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:37,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:37,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:37,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:37,783 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:37,784 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,784 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,785 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:37,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:37,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:37,785 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:37,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:37,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:37,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:37,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:37,786 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:37,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:37,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:37,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,810 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,810 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:37,810 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:37,810 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:37,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:37,810 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:37,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:37,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:37,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:37,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:37,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:37,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:37,810 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:37,810 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,811 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:37,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:37,811 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,811 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,811 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:37,813 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:54:37,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:37,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:37,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:37,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:37,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:37,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:37,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095277838, max timestamp=1681095277851Num uniques 859; 2023-04-10 02:54:37,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:54:37,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095277838, max timestamp=1681095277851Num uniques 859; 2023-04-10 02:54:37,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095277838, max timestamp=1681095277851Num uniques 859; 2023-04-10 02:54:37,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:54:37,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:54:37,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:37,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095337878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:37,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095337879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,911 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:37,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:37,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:37,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:37,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:37,914 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:37,914 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:37,914 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:37,914 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:37,915 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,915 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:37,916 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:37,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:37,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:37,917 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,917 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:37,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:37,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:37,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:37,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:37,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:37,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:37,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:37,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:37,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:37,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:37,918 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:37,918 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:37,918 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:37,918 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:37,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:37,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:37,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:37,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:37,920 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,920 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:37,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:37,920 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,920 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:37,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:37,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@abcf78[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:37,922 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:37,922 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,923 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,923 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:37,923 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:37,923 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:37,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:37,924 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:37,924 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:37,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:37,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:37,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:37,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:37,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:37,927 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:37,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:37,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:37,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:37,928 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:37,930 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,930 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,930 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:37,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:37,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:37,932 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,934 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:37,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:37,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:37,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:37,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,936 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:37,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:37,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:37,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:37,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:37,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:37,937 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:37,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:37,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:37,940 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:37,940 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:37,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:37,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095337982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:37,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:37,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095337983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,040 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:38,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:38,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:38,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:38,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:38,044 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:38,044 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,044 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:38,044 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:38,044 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,044 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,046 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:38,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:38,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:38,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:38,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:38,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:38,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:38,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:38,048 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,048 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:38,048 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:38,048 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:38,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:38,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:38,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:38,053 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:38,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ca9a850[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:38,055 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:38,055 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,056 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:38,057 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,057 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:38,058 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,058 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:38,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,058 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,058 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:38,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:38,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:38,064 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:38,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:38,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:38,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:38,064 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:38,066 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,066 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,066 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,068 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,069 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,073 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,073 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:38,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:38,074 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,074 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:38,074 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:38,174 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:38,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:38,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:38,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:38,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:38,178 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:38,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:38,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:38,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,181 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,181 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,181 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,181 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:38,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:38,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:38,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:38,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:38,182 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:38,182 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,182 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:38,182 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:38,182 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:38,182 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,182 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:38,183 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:38,183 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:38,184 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,184 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,185 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,185 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:38,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf54e83[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:38,186 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:38,186 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,187 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:38,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095338186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,188 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,188 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:38,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:38,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095338186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,189 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:38,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:38,191 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:38,191 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:38,191 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:38,191 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:38,191 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:38,194 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,194 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,194 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,195 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:38,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:38,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,198 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:38,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,198 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:38,198 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:38,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,249 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410fab5de39edaa44eb91f9508448d18e33 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fab5de39edaa44eb91f9508448d18e33 2023-04-10 02:54:38,298 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:38,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:38,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:38,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:38,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:38,301 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:38,302 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,302 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:38,302 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:38,302 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,302 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,303 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,303 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,304 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,304 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:38,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:38,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:38,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:38,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:38,305 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,305 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:38,305 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:38,305 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:38,306 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:38,310 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:38,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f9e48fc[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:38,312 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:38,312 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,313 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,314 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:38,314 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:38,314 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:38,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:38,315 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:38,315 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,315 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,315 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:38,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:38,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:38,318 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:38,319 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:38,319 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:38,319 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:38,319 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:38,320 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,320 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,320 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,324 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,332 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,336 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:38,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:38,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,337 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:38,337 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:38,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,437 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:38,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:38,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:38,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:38,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:38,442 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:38,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:38,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:38,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,444 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,444 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,444 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,444 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:38,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:38,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:38,448 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,448 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:38,448 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:38,448 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,449 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:38,449 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:38,449 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,449 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:38,449 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:38,451 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:38,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:38,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31908a78[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:38,453 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:38,453 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,455 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,455 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:38,455 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:38,455 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:38,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:38,457 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,457 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:38,457 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,457 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:38,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:38,460 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:38,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:38,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:38,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:38,460 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:38,462 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,462 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,462 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,463 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,464 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,466 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,466 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:38,466 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,466 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,466 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:38,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:38,466 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,467 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,467 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,467 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,467 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,467 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,467 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:38,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:38,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095338488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:38,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095338491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,567 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:38,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:38,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:38,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:38,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:38,570 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:38,570 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,570 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:38,570 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:38,571 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,571 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,572 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:38,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:38,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:38,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:38,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:38,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:38,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:38,574 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,574 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:38,575 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:38,575 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:38,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:38,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:38,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:38,579 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:38,580 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:38,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@509a4f32[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:38,580 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,581 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,582 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:38,582 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:38,582 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,582 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:38,582 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:38,583 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:38,583 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,583 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,583 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:38,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:38,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:38,586 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:38,586 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:38,586 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:38,586 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:38,586 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:38,587 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,587 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,587 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,588 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,589 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,592 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:38,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:38,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:38,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,592 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:38,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,592 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:38,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,656 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=282, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/b7f2cf700fce4c3bb25e173b5849a8bf 2023-04-10 02:54:38,692 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:38,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:38,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:38,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:38,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:38,695 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:38,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:38,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:38,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,697 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,698 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,698 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:38,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:38,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:38,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:38,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:38,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:38,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:38,699 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,699 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:38,699 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:38,699 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:38,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:38,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:38,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:38,701 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,701 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,701 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,701 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:38,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:38,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@266d82f4[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:38,702 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:38,703 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,707 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:38,707 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:38,707 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:38,707 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,707 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:38,708 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:38,708 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,709 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,709 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:38,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:38,711 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:38,711 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:38,711 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:38,711 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:38,711 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:38,712 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,712 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,712 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,713 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,714 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,717 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,717 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,717 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,717 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:38,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:38,717 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,717 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,717 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,717 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,717 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:38,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,717 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,717 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,717 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:38,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:38,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,818 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:38,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:38,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:38,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:38,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:38,821 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:38,821 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,821 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:38,821 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:38,822 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,822 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,824 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,824 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,824 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,824 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:38,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:38,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:38,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:38,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:38,825 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:38,826 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,826 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:38,826 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:38,826 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:38,826 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,826 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:38,826 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:38,826 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,826 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:38,826 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:38,826 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,826 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:38,826 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:38,829 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,829 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,830 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,830 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:38,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:38,832 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:38,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ba9e76d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:38,832 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,833 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:38,833 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:38,833 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,833 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:38,833 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:38,834 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:38,834 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,834 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,834 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:38,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:38,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:38,837 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:38,837 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:38,837 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:38,837 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:38,837 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:38,839 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,839 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,839 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,840 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,841 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,844 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,844 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,844 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,844 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:38,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:38,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:38,844 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,844 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,844 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,844 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,844 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,844 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:38,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,844 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:38,844 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,944 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:38,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:38,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:38,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:38,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:38,948 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:38,948 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,948 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:38,948 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:38,949 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,949 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,950 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:38,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:38,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:38,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:38,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:38,951 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:38,951 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:38,951 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:38,952 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:38,952 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:38,952 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,952 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:38,952 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:38,952 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:38,952 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:38,952 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:38,952 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,952 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:38,952 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:38,953 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,953 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,954 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,954 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:38,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:38,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@edf792a[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:38,955 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:38,955 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,957 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,958 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:38,958 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:38,958 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:38,958 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,958 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:38,959 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:38,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,959 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,959 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:38,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:38,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:38,961 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:38,961 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:38,961 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:38,962 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:38,962 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:38,963 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,963 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,963 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:38,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,963 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:38,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:38,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,965 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:38,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:38,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,966 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,967 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,967 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:38,967 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:38,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:38,967 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:38,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:38,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:38,967 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,967 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,967 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:38,967 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:38,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:38,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:38,967 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:38,967 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:38,967 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:38,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:38,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:38,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:38,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095338992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:38,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:38,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095338994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/d64513d479ea4e62b712eae537fa093a 2023-04-10 02:54:39,067 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:39,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:39,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:39,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:39,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:39,075 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:39,075 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:39,075 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:39,075 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:39,076 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:39,076 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:39,078 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,078 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:39,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:39,078 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,079 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:39,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:39,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:39,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:39,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:39,080 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:39,080 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:39,081 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:39,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:39,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:39,087 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:39,087 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:39,087 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:39,087 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:39,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:39,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:39,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:39,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:39,088 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/225a3a7b35284dcbae1071191159ff03 2023-04-10 02:54:39,089 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:39,089 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:39,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:39,089 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:39,089 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:39,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:39,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:39,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:39,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:39,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:39,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c3e441c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:39,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:39,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:39,093 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:39,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:39,094 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:39,094 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:39,094 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,094 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:39,094 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:39,095 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/b7f2cf700fce4c3bb25e173b5849a8bf as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b7f2cf700fce4c3bb25e173b5849a8bf 2023-04-10 02:54:39,099 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:39,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:39,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:39,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:39,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:39,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b7f2cf700fce4c3bb25e173b5849a8bf, entries=350, sequenceid=282, filesize=52.9 K 2023-04-10 02:54:39,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:39,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/d64513d479ea4e62b712eae537fa093a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d64513d479ea4e62b712eae537fa093a 2023-04-10 02:54:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:39,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:39,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:39,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:39,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:39,104 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:39,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:39,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:39,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:39,105 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:39,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:39,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:39,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:39,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:39,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:39,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:39,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:39,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:39,107 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:39,107 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d64513d479ea4e62b712eae537fa093a, entries=150, sequenceid=282, filesize=11.9 K 2023-04-10 02:54:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:39,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,109 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/225a3a7b35284dcbae1071191159ff03 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/225a3a7b35284dcbae1071191159ff03 2023-04-10 02:54:39,109 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:39,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:39,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,112 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:39,112 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:39,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:39,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:39,112 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,112 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:39,112 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:39,112 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:39,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:39,112 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:39,113 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:39,112 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:39,113 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:39,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:39,113 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:39,113 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:39,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/225a3a7b35284dcbae1071191159ff03, entries=150, sequenceid=282, filesize=11.9 K 2023-04-10 02:54:39,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a944754ff766fcb82f3c238c462e40cc in 1301ms, sequenceid=282, compaction requested=true 2023-04-10 02:54:39,114 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:39,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:39,115 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-10 02:54:39,115 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-10 02:54:39,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:39,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:39,117 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163266 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-10 02:54:39,117 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/A is initiating minor compaction (all files) 2023-04-10 02:54:39,117 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 65746 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-10 02:54:39,117 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/A in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:39,117 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/B is initiating minor compaction (all files) 2023-04-10 02:54:39,117 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/da0c282a0c52472f94357fe89467e9cc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/a44cccd818c94493af87e52eb556b234, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/9e516e3a294b46cc8fbb9c92f733406f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/2124b1fd6cac4205a7687a0f14b0a81a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/05bfb749a70240569a37492d5e60d008, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b7f2cf700fce4c3bb25e173b5849a8bf] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=159.4 K 2023-04-10 02:54:39,117 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/B in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:39,117 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/115c5078929646e9b021db28cfbd47e8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5e9fce30affb45dab1ff7627bc2cc85b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5aca45fb63ab47c3b6b56656b3001417, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d056f82b3e7143bf9e8b4e77dace8fcf, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/025003e13fe24b22a946b475dcfb4d23, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d64513d479ea4e62b712eae537fa093a] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=64.2 K 2023-04-10 02:54:39,118 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting da0c282a0c52472f94357fe89467e9cc, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681095272519 2023-04-10 02:54:39,118 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting a44cccd818c94493af87e52eb556b234, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681095273130 2023-04-10 02:54:39,118 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 115c5078929646e9b021db28cfbd47e8, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681095272519 2023-04-10 02:54:39,118 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 9e516e3a294b46cc8fbb9c92f733406f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681095274325 2023-04-10 02:54:39,118 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 5e9fce30affb45dab1ff7627bc2cc85b, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681095273130 2023-04-10 02:54:39,119 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 2124b1fd6cac4205a7687a0f14b0a81a, keycount=50, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681095275450 2023-04-10 02:54:39,119 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 5aca45fb63ab47c3b6b56656b3001417, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681095274325 2023-04-10 02:54:39,119 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 05bfb749a70240569a37492d5e60d008, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681095275530 2023-04-10 02:54:39,119 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d056f82b3e7143bf9e8b4e77dace8fcf, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681095275450 2023-04-10 02:54:39,119 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting b7f2cf700fce4c3bb25e173b5849a8bf, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1681095276653 2023-04-10 02:54:39,119 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 025003e13fe24b22a946b475dcfb4d23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681095275530 2023-04-10 02:54:39,120 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d64513d479ea4e62b712eae537fa093a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1681095277778 2023-04-10 02:54:39,141 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#B#compaction#181 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:39,147 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#A#compaction#180 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:39,213 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:39,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:39,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:39,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:39,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:39,216 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:39,216 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:39,216 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:39,216 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:39,217 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:39,217 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:39,218 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:39,218 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:39,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:39,218 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:39,218 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:39,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:39,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:39,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:39,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:39,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:39,219 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:39,219 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:39,219 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:39,219 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:39,219 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:39,220 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:39,220 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:39,220 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-10 02:54:39,220 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:39,220 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:39,220 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:39,220 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:39,220 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:39,220 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:39,570 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/2fc175114a2649c882f141f0ffdb5ab2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/2fc175114a2649c882f141f0ffdb5ab2 2023-04-10 02:54:39,575 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/665f18ea269940b48749adc683d926cd as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/665f18ea269940b48749adc683d926cd 2023-04-10 02:54:39,576 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/B of a944754ff766fcb82f3c238c462e40cc into 2fc175114a2649c882f141f0ffdb5ab2(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:39,576 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:39,576 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/B, priority=10, startTime=1681095279115; duration=0sec 2023-04-10 02:54:39,576 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:39,576 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-10 02:54:39,578 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 65746 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-10 02:54:39,578 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/C is initiating minor compaction (all files) 2023-04-10 02:54:39,578 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/C in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:39,579 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/bcdfd786491f44bfadaa2f211a79b67d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/38edeac5f2dd41d09cefd9caf08b15f5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/6be3456d2b7542d29253bb8e96f877c7, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/55f7f1cc13b44a7b824acc35d4680296, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/57eb9cfbe2104822ab24995b451afb90, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/225a3a7b35284dcbae1071191159ff03] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=64.2 K 2023-04-10 02:54:39,579 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting bcdfd786491f44bfadaa2f211a79b67d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681095272519 2023-04-10 02:54:39,579 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 38edeac5f2dd41d09cefd9caf08b15f5, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681095273130 2023-04-10 02:54:39,580 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6be3456d2b7542d29253bb8e96f877c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681095274325 2023-04-10 02:54:39,580 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 55f7f1cc13b44a7b824acc35d4680296, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681095275450 2023-04-10 02:54:39,580 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 57eb9cfbe2104822ab24995b451afb90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681095275530 2023-04-10 02:54:39,581 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 225a3a7b35284dcbae1071191159ff03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1681095277778 2023-04-10 02:54:39,581 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/A of a944754ff766fcb82f3c238c462e40cc into 665f18ea269940b48749adc683d926cd(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:39,581 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:39,581 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/A, priority=10, startTime=1681095279115; duration=0sec 2023-04-10 02:54:39,581 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:39,590 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#C#compaction#183 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:39,635 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410c291308d03b640278e00c01a358e2a6f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410c291308d03b640278e00c01a358e2a6f 2023-04-10 02:54:39,640 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=318, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/1459e19a118e4205bb481e758627902b 2023-04-10 02:54:40,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:40,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:40,005 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/1e4bc142ae444301bca8e3a30435c885 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/1e4bc142ae444301bca8e3a30435c885 2023-04-10 02:54:40,010 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/C of a944754ff766fcb82f3c238c462e40cc into 1e4bc142ae444301bca8e3a30435c885(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:40,010 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:40,010 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/C, priority=10, startTime=1681095279115; duration=0sec 2023-04-10 02:54:40,010 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:40,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:40,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095340013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:40,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095340016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,051 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/7e3e82b2c2be4bdbbf97b61a4f6e7836 2023-04-10 02:54:40,067 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/df87cff819904432bfa05e49ad2863e5 2023-04-10 02:54:40,072 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/1459e19a118e4205bb481e758627902b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/1459e19a118e4205bb481e758627902b 2023-04-10 02:54:40,076 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/1459e19a118e4205bb481e758627902b, entries=150, sequenceid=318, filesize=25.4 K 2023-04-10 02:54:40,077 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/7e3e82b2c2be4bdbbf97b61a4f6e7836 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/7e3e82b2c2be4bdbbf97b61a4f6e7836 2023-04-10 02:54:40,080 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/7e3e82b2c2be4bdbbf97b61a4f6e7836, entries=150, sequenceid=318, filesize=11.9 K 2023-04-10 02:54:40,081 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/df87cff819904432bfa05e49ad2863e5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/df87cff819904432bfa05e49ad2863e5 2023-04-10 02:54:40,086 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/df87cff819904432bfa05e49ad2863e5, entries=150, sequenceid=318, filesize=11.9 K 2023-04-10 02:54:40,086 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for a944754ff766fcb82f3c238c462e40cc in 866ms, sequenceid=318, compaction requested=false 2023-04-10 02:54:40,087 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:40,087 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:40,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:40,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:40,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:40,089 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,089 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,090 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,090 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:40,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@468e1d4d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:40,095 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:40,095 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,096 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,096 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:40,096 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:40,096 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:40,097 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,097 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:40,097 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:40,097 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,097 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,097 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:40,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:40,101 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:40,101 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:40,101 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:40,101 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:40,101 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:40,103 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,103 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,104 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,105 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,106 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:40,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:40,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,110 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:40,110 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:40,110 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:40,118 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:54:40,119 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:40,119 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:40,119 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:40,119 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:40,119 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:40,119 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:40,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:40,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095340149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:40,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095340153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,210 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:40,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:40,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:40,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:40,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:40,213 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:40,213 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,213 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:40,213 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:40,214 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,214 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,216 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,216 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,216 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,216 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:40,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:40,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:40,218 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:40,218 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,218 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:40,218 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:40,218 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:40,218 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,218 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:40,218 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:40,218 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,218 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:40,218 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:40,219 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,219 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:40,219 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:40,220 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,220 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,221 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,221 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:40,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32e8187e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:40,222 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:40,222 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,223 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,223 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,223 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:40,223 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:40,223 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:40,223 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:40,226 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:40,226 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,226 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,226 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:40,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:40,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:40,229 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:40,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:40,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:40,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:40,229 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:40,230 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,230 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,230 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,232 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,234 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,234 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,234 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,234 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,234 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:40,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:40,234 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,234 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,234 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,234 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:40,234 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:40,234 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,235 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:40,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095340253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:40,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095340257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,335 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:40,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:40,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:40,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:40,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:40,341 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:40,341 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,342 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:40,342 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:40,342 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,342 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,343 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,343 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,345 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,345 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:40,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:40,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:40,353 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:40,353 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,353 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:40,353 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:40,353 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:40,353 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,353 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:40,353 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:40,353 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,353 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:40,353 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:40,353 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,354 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:40,354 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:40,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,357 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,357 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,357 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:40,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:40,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@470b2120[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:40,358 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:40,358 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,359 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,360 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:40,360 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:40,360 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,360 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:40,360 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:40,362 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,362 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:40,362 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,362 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:40,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:40,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:40,365 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:40,365 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:40,365 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:40,365 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:40,365 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:40,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,368 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,369 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,375 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:40,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,376 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:40,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:40,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,376 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,376 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,376 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:40,376 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:40,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:40,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095340458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:40,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095340459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,476 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:40,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:40,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:40,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:40,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:40,479 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:40,479 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,479 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:40,479 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:40,480 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,480 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,481 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,481 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,481 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,482 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:40,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:40,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:40,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:40,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:40,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:40,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:40,483 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,483 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:40,483 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:40,483 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:40,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:40,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:40,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:40,485 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,485 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,485 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,485 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:40,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:40,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39eb8379[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:40,487 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:40,487 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,488 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,488 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:40,488 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:40,488 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:40,488 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,488 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:40,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:40,495 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:40,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:40,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:40,497 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:40,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:40,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:40,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:40,497 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:40,499 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,499 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,499 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,499 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,500 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,503 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,503 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,503 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,503 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,503 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:40,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:40,503 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,503 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,503 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,503 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:40,503 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,503 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,503 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:40,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,554 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410e836275427084327bd7fac5ea785aa08 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410e836275427084327bd7fac5ea785aa08 2023-04-10 02:54:40,603 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:40,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:40,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:40,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:40,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:40,607 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:40,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:40,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:40,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,608 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,609 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,609 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:40,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:40,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:40,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:40,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:40,610 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:40,610 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,610 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:40,610 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:40,610 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:40,610 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,610 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:40,611 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:40,611 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:40,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:40,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:40,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:40,613 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,613 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,613 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,614 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:40,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f39957c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:40,615 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:40,615 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,616 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,616 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,616 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:40,618 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:40,618 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:40,618 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:40,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:40,619 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:40,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:40,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:40,621 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:40,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:40,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:40,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:40,622 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:40,623 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,623 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,623 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,625 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,626 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:40,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:40,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,631 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,631 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,630 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,631 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,631 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:40,631 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:40,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:40,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:40,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53902 deadline: 1681095340705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,707 DEBUG [Thread-945] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8184 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:54:40,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:40,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53910 deadline: 1681095340707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,710 DEBUG [Thread-941] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8188 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:54:40,731 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:40,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:40,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:40,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:40,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:40,737 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:40,737 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,737 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:40,737 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:40,737 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,737 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:40,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:40,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:40,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:40,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:40,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:40,741 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:40,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:40,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:40,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,741 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:40,741 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:40,741 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,741 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:40,741 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:40,742 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,742 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,743 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,743 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:40,745 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:40,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d60470d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:40,745 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,746 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:40,746 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:40,746 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:40,746 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,746 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:40,752 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,752 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,752 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,752 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:40,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:40,754 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:40,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:40,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:40,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:40,754 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:40,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,756 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,758 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,762 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,762 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,762 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,762 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,762 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,762 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,762 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,762 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,762 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:40,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:40,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:40,762 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,763 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:40,763 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:40,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:40,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095340764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:40,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095340764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,863 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:40,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:40,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:40,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:40,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:40,866 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:40,866 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,866 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:40,866 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:40,866 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,866 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,869 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,869 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,869 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,869 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:40,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:40,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:40,870 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:40,870 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,870 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:40,870 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:40,870 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:40,870 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,871 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:40,871 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:40,871 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,871 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:40,871 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:40,871 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,871 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:40,871 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:40,872 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,872 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,872 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,873 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:40,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63fc7ba0[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:40,875 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:40,875 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,876 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,876 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:40,876 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:40,876 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,876 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:40,876 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:40,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:40,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,878 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:40,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:40,880 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:40,880 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:40,880 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:40,880 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:40,880 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:40,882 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,882 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,882 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,882 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,884 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:40,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,886 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,886 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,886 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,886 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:40,886 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,886 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:40,886 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,886 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,886 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:40,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:40,886 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,887 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:40,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:40,887 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:40,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:40,960 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=333, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/6d4843b28c324e3b8910ecc1dc20e351 2023-04-10 02:54:40,975 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/75eab5e2c488439088b0b221be88a022 2023-04-10 02:54:40,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/8f1d16836dce4ebb90c25dde6fb72107 2023-04-10 02:54:40,987 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:40,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:40,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:40,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:40,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:40,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/6d4843b28c324e3b8910ecc1dc20e351 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/6d4843b28c324e3b8910ecc1dc20e351 2023-04-10 02:54:40,991 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:40,991 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,991 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:40,991 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:40,992 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,992 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,993 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:40,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:40,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:40,994 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,994 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,994 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:40,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:40,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:40,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:40,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:40,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:40,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:40,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:40,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:40,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:40,995 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,995 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/6d4843b28c324e3b8910ecc1dc20e351, entries=150, sequenceid=333, filesize=25.4 K 2023-04-10 02:54:40,995 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:40,996 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:40,996 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:40,996 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:40,996 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:40,996 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,996 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:40,996 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:40,996 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/75eab5e2c488439088b0b221be88a022 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/75eab5e2c488439088b0b221be88a022 2023-04-10 02:54:40,997 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,997 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:40,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:40,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:40,998 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:40,998 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:40,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:40,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:40,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:40,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:41,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:41,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@669cced[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:41,000 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:41,000 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:41,000 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/75eab5e2c488439088b0b221be88a022, entries=150, sequenceid=333, filesize=11.9 K 2023-04-10 02:54:41,001 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/8f1d16836dce4ebb90c25dde6fb72107 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/8f1d16836dce4ebb90c25dde6fb72107 2023-04-10 02:54:41,002 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:41,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:41,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:41,002 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:41,002 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:41,002 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:41,003 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,003 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:41,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,004 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:41,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:41,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:41,004 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:41,004 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:41,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:41,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:41,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:41,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:41,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:41,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:41,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:41,007 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:41,007 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:41,007 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:41,007 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:41,007 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:41,009 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:41,009 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:41,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:41,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:41,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:41,009 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:41,009 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/8f1d16836dce4ebb90c25dde6fb72107, entries=150, sequenceid=333, filesize=11.9 K 2023-04-10 02:54:41,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:41,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:41,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:41,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:41,010 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,010 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a944754ff766fcb82f3c238c462e40cc in 892ms, sequenceid=333, compaction requested=true 2023-04-10 02:54:41,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:41,010 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:41,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:41,010 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:41,010 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:41,010 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:41,010 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:41,011 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:41,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:41,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:41,012 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:41,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:41,012 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/B is initiating minor compaction (all files) 2023-04-10 02:54:41,012 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/B in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:41,012 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78752 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:41,012 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,012 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/2fc175114a2649c882f141f0ffdb5ab2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/7e3e82b2c2be4bdbbf97b61a4f6e7836, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/75eab5e2c488439088b0b221be88a022] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=36.3 K 2023-04-10 02:54:41,012 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/A is initiating minor compaction (all files) 2023-04-10 02:54:41,013 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/A in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:41,013 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/665f18ea269940b48749adc683d926cd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/1459e19a118e4205bb481e758627902b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/6d4843b28c324e3b8910ecc1dc20e351] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=76.9 K 2023-04-10 02:54:41,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,013 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 2fc175114a2649c882f141f0ffdb5ab2, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1681095277778 2023-04-10 02:54:41,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:41,013 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 665f18ea269940b48749adc683d926cd, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1681095277778 2023-04-10 02:54:41,014 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 7e3e82b2c2be4bdbbf97b61a4f6e7836, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681095277872 2023-04-10 02:54:41,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:41,014 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 1459e19a118e4205bb481e758627902b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681095277872 2023-04-10 02:54:41,014 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 75eab5e2c488439088b0b221be88a022, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1681095280012 2023-04-10 02:54:41,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,014 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 6d4843b28c324e3b8910ecc1dc20e351, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1681095280012 2023-04-10 02:54:41,021 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:41,021 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:41,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:41,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:41,021 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,021 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:41,021 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:41,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:41,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:41,021 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:41,021 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,021 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:41,021 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:41,021 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:41,021 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:41,022 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:41,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:41,031 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#B#compaction#190 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:41,034 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#A#compaction#189 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:41,055 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/e6e2a09449da41a3acaa789ada9c12a5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/e6e2a09449da41a3acaa789ada9c12a5 2023-04-10 02:54:41,061 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/B of a944754ff766fcb82f3c238c462e40cc into e6e2a09449da41a3acaa789ada9c12a5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:41,061 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:41,061 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/B, priority=13, startTime=1681095281010; duration=0sec 2023-04-10 02:54:41,061 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:41,061 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:41,062 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:41,062 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): a944754ff766fcb82f3c238c462e40cc/C is initiating minor compaction (all files) 2023-04-10 02:54:41,062 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a944754ff766fcb82f3c238c462e40cc/C in TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:41,062 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/1e4bc142ae444301bca8e3a30435c885, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/df87cff819904432bfa05e49ad2863e5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/8f1d16836dce4ebb90c25dde6fb72107] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp, totalSize=36.3 K 2023-04-10 02:54:41,063 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 1e4bc142ae444301bca8e3a30435c885, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1681095277778 2023-04-10 02:54:41,063 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting df87cff819904432bfa05e49ad2863e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681095277872 2023-04-10 02:54:41,064 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 8f1d16836dce4ebb90c25dde6fb72107, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1681095280012 2023-04-10 02:54:41,070 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): a944754ff766fcb82f3c238c462e40cc#C#compaction#191 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:41,086 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/c52d4710e73b4cb99679f0fe147c5a42 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/c52d4710e73b4cb99679f0fe147c5a42 2023-04-10 02:54:41,091 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/C of a944754ff766fcb82f3c238c462e40cc into c52d4710e73b4cb99679f0fe147c5a42(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:41,091 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:41,091 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/C, priority=13, startTime=1681095281011; duration=0sec 2023-04-10 02:54:41,091 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:41,122 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:41,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:41,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:41,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:41,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:41,125 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:41,125 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:41,125 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:41,125 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:41,126 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:41,126 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:41,129 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:41,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:41,130 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,130 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:41,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:41,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:41,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:41,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:41,131 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:41,131 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:41,131 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:41,131 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:41,131 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:41,131 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:41,131 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:41,131 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-10 02:54:41,131 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:41,131 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:41,131 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:41,131 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:41,131 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:41,132 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:41,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:41,268 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:41,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:41,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095341279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:41,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095341282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:41,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095341382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:41,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095341385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,553 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410e735f45560094bc48bb023b677b2a967 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410e735f45560094bc48bb023b677b2a967 2023-04-10 02:54:41,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:41,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095341586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:41,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095341589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,868 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/e1e11a7743a14458b314da05662953ca as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/e1e11a7743a14458b314da05662953ca 2023-04-10 02:54:41,873 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a944754ff766fcb82f3c238c462e40cc/A of a944754ff766fcb82f3c238c462e40cc into e1e11a7743a14458b314da05662953ca(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:41,873 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:41,873 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc., storeName=a944754ff766fcb82f3c238c462e40cc/A, priority=13, startTime=1681095281010; duration=0sec 2023-04-10 02:54:41,873 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:41,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:41,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095341888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:41,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095341890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:41,961 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=355, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/21cd005c36a24418bed04765994b8032 2023-04-10 02:54:42,378 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/d93bcd5394c1442f8346256a8a5ed8f7 2023-04-10 02:54:42,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:42,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53894 deadline: 1681095342392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:42,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53892 deadline: 1681095342392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,796 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/b9e0a9453fa24d499b37be4c575d73a4 2023-04-10 02:54:42,800 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/21cd005c36a24418bed04765994b8032 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/21cd005c36a24418bed04765994b8032 2023-04-10 02:54:42,804 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/21cd005c36a24418bed04765994b8032, entries=150, sequenceid=355, filesize=25.4 K 2023-04-10 02:54:42,805 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/d93bcd5394c1442f8346256a8a5ed8f7 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d93bcd5394c1442f8346256a8a5ed8f7 2023-04-10 02:54:42,809 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d93bcd5394c1442f8346256a8a5ed8f7, entries=150, sequenceid=355, filesize=11.9 K 2023-04-10 02:54:42,810 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/b9e0a9453fa24d499b37be4c575d73a4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/b9e0a9453fa24d499b37be4c575d73a4 2023-04-10 02:54:42,814 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/b9e0a9453fa24d499b37be4c575d73a4, entries=150, sequenceid=355, filesize=11.9 K 2023-04-10 02:54:42,815 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a944754ff766fcb82f3c238c462e40cc in 1684ms, sequenceid=355, compaction requested=false 2023-04-10 02:54:42,815 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:42,815 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:42,817 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,817 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:42,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:42,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:42,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:42,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45b82b29[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:42,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:42,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:42,821 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:42,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:42,821 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,822 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:42,822 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:42,822 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:42,822 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:42,822 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:42,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:42,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:42,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:42,825 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:42,825 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:42,825 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:42,825 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:42,825 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:42,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:42,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:42,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:42,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:42,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:42,829 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:42,830 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:42,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:42,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:42,832 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:42,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:42,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:42,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:42,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:42,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:42,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:42,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:42,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:42,833 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:42,833 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:42,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:42,933 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:42,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:42,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:42,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:42,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:42,937 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:42,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:42,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:42,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:42,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:42,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:42,941 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,941 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:42,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:42,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:42,942 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:42,942 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:42,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:42,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:42,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:42,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:42,943 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:42,943 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:42,943 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:42,943 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:42,943 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:42,943 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:42,943 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. started... 2023-04-10 02:54:42,943 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:54:42,943 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:42,944 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:42,944 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:42,944 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:42,944 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:42,944 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:43,302 DEBUG [Thread-948] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54451f11 to 127.0.0.1:60568 2023-04-10 02:54:43,302 DEBUG [Thread-948] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:43,303 DEBUG [Thread-950] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:60568 2023-04-10 02:54:43,303 DEBUG [Thread-950] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:43,303 DEBUG [Thread-952] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:60568 2023-04-10 02:54:43,303 DEBUG [Thread-952] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:43,305 DEBUG [Thread-954] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3bb940 to 127.0.0.1:60568 2023-04-10 02:54:43,305 DEBUG [Thread-954] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:43,357 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410e44b588ad30b4554a4c6dc0a3a0d88cb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410e44b588ad30b4554a4c6dc0a3a0d88cb 2023-04-10 02:54:43,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:43,396 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. as already flushing 2023-04-10 02:54:43,396 DEBUG [Thread-939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1aa5eef5 to 127.0.0.1:60568 2023-04-10 02:54:43,396 DEBUG [Thread-939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:43,400 DEBUG [Thread-943] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69f15cb4 to 127.0.0.1:60568 2023-04-10 02:54:43,400 DEBUG [Thread-943] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:43,762 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=368, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/d70ada5dd9bb488ab32d81bc26ea694c 2023-04-10 02:54:44,172 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/8fc1584ea3414d3c852c3f37d34360cc 2023-04-10 02:54:44,248 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-10 02:54:44,548 DEBUG [Thread-937] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b3934fb to 127.0.0.1:60568 2023-04-10 02:54:44,548 DEBUG [Thread-937] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:44,583 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/79f3481d13674d98b2f5c01137239499 2023-04-10 02:54:44,589 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/d70ada5dd9bb488ab32d81bc26ea694c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/d70ada5dd9bb488ab32d81bc26ea694c 2023-04-10 02:54:44,593 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/d70ada5dd9bb488ab32d81bc26ea694c, entries=150, sequenceid=368, filesize=25.4 K 2023-04-10 02:54:44,593 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/8fc1584ea3414d3c852c3f37d34360cc as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/8fc1584ea3414d3c852c3f37d34360cc 2023-04-10 02:54:44,597 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/8fc1584ea3414d3c852c3f37d34360cc, entries=150, sequenceid=368, filesize=11.9 K 2023-04-10 02:54:44,598 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/79f3481d13674d98b2f5c01137239499 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/79f3481d13674d98b2f5c01137239499 2023-04-10 02:54:44,602 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/79f3481d13674d98b2f5c01137239499, entries=150, sequenceid=368, filesize=11.9 K 2023-04-10 02:54:44,603 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=20.13 KB/20610 for a944754ff766fcb82f3c238c462e40cc in 1660ms, sequenceid=368, compaction requested=true 2023-04-10 02:54:44,603 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:44,603 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:44,603 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:44,603 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:44,603 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,603 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:44,603 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:44,604 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:44,604 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:44,605 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:44,605 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:44,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:44,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a1f54bc[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:44,606 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:44,606 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:44,608 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:44,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:44,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:44,608 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:44,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:44,608 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:44,608 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:44,610 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:44,610 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,610 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:44,610 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:44,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:44,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:44,612 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:44,612 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:44,612 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:44,612 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:44,612 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:44,613 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:44,613 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:44,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:44,614 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:44,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:44,614 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:44,615 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,617 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,617 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:44,617 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:44,617 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:44,617 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:44,617 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:44,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:44,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:44,617 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:44,617 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:44,617 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:44,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:44,618 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:44,618 DEBUG [Thread-947] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:44,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:44,618 DEBUG [Thread-947] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:44,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:44,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:44,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:44,718 DEBUG [Thread-947] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:44,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:50,737 DEBUG [Thread-941] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1441c9f0 to 127.0.0.1:60568 2023-04-10 02:54:50,737 DEBUG [Thread-941] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:50,793 DEBUG [Thread-945] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c1c92c6 to 127.0.0.1:60568 2023-04-10 02:54:50,793 DEBUG [Thread-945] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:50,793 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-10 02:54:50,793 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 14 2023-04-10 02:54:50,793 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-04-10 02:54:50,793 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 37 2023-04-10 02:54:50,793 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 107 2023-04-10 02:54:50,793 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 37 2023-04-10 02:54:50,794 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-10 02:54:50,794 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 5248 2023-04-10 02:54:50,794 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 5165 2023-04-10 02:54:50,794 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-10 02:54:50,794 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 2197 2023-04-10 02:54:50,794 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 6591 rows 2023-04-10 02:54:50,794 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 2213 2023-04-10 02:54:50,794 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 6633 rows 2023-04-10 02:54:50,794 INFO [Listener at localhost/44533] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-10 02:54:50,794 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x33c943f0 to 127.0.0.1:60568 2023-04-10 02:54:50,794 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:54:50,797 INFO [Listener at localhost/44533] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-10 02:54:50,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-10 02:54:50,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:50,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-10 02:54:50,802 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095290801"}]},"ts":"1681095290801"} 2023-04-10 02:54:50,802 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-10 02:54:50,805 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-10 02:54:50,805 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a944754ff766fcb82f3c238c462e40cc, UNASSIGN}] 2023-04-10 02:54:50,806 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a944754ff766fcb82f3c238c462e40cc, UNASSIGN 2023-04-10 02:54:50,807 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=a944754ff766fcb82f3c238c462e40cc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:50,807 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095290807"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095290807"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095290807"}]},"ts":"1681095290807"} 2023-04-10 02:54:50,808 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:54:50,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-10 02:54:50,960 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a944754ff766fcb82f3c238c462e40cc, disabling compactions & flushes 2023-04-10 02:54:50,960 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. after waiting 0 ms 2023-04-10 02:54:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:50,960 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing a944754ff766fcb82f3c238c462e40cc 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-10 02:54:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=A 2023-04-10 02:54:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=B 2023-04-10 02:54:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a944754ff766fcb82f3c238c462e40cc, store=C 2023-04-10 02:54:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:51,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-10 02:54:51,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041023d067f07a2e4bb693644e7e5a81b0fa to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041023d067f07a2e4bb693644e7e5a81b0fa 2023-04-10 02:54:51,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-10 02:54:51,779 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=376, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/4f3511ec093346e1a879797f1f5de624 2023-04-10 02:54:51,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-10 02:54:52,188 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/001d2ed1fec94c2bb3cc0bd386e40ecc 2023-04-10 02:54:52,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/98ca06e4a1034b4491b087222c0d9bb4 2023-04-10 02:54:52,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/A/4f3511ec093346e1a879797f1f5de624 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/4f3511ec093346e1a879797f1f5de624 2023-04-10 02:54:52,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/4f3511ec093346e1a879797f1f5de624, entries=100, sequenceid=376, filesize=18.6 K 2023-04-10 02:54:52,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/B/001d2ed1fec94c2bb3cc0bd386e40ecc as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/001d2ed1fec94c2bb3cc0bd386e40ecc 2023-04-10 02:54:52,610 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/001d2ed1fec94c2bb3cc0bd386e40ecc, entries=100, sequenceid=376, filesize=9.5 K 2023-04-10 02:54:52,611 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/.tmp/C/98ca06e4a1034b4491b087222c0d9bb4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/98ca06e4a1034b4491b087222c0d9bb4 2023-04-10 02:54:52,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/98ca06e4a1034b4491b087222c0d9bb4, entries=100, sequenceid=376, filesize=9.5 K 2023-04-10 02:54:52,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for a944754ff766fcb82f3c238c462e40cc in 1654ms, sequenceid=376, compaction requested=true 2023-04-10 02:54:52,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/2f1a2c282e2643a3af0b9a3b1d4e3902, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/c943a7bb010d490b947669e76977f91b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/e4952c6765614d629fc9be85dd4baf67, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/26e508a7ec2a4823abd248d565e5eaaa, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/51a235e84cef45449ae69d492fbdb4dd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/98d5ed57ccee439db0667428fe6c9665, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b180dd1bf8474d4db026fc4c1e56bf0e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/73ada6a07dcb49f1a6b55ed7da880bc7, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b002b2e7673b4245bc0f5886ee2038c1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/ee6db7ce430b4a738df078ac0cc3e647, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b2a9ba1195944567ad0563b712e064eb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/34802c322c824c9b8890ccd396140d09, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/da0c282a0c52472f94357fe89467e9cc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/a44cccd818c94493af87e52eb556b234, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/9e516e3a294b46cc8fbb9c92f733406f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/2124b1fd6cac4205a7687a0f14b0a81a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/05bfb749a70240569a37492d5e60d008, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b7f2cf700fce4c3bb25e173b5849a8bf, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/665f18ea269940b48749adc683d926cd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/1459e19a118e4205bb481e758627902b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/6d4843b28c324e3b8910ecc1dc20e351] to archive 2023-04-10 02:54:52,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:54:52,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/2f1a2c282e2643a3af0b9a3b1d4e3902 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/2f1a2c282e2643a3af0b9a3b1d4e3902 2023-04-10 02:54:52,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/c943a7bb010d490b947669e76977f91b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/c943a7bb010d490b947669e76977f91b 2023-04-10 02:54:52,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/e4952c6765614d629fc9be85dd4baf67 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/e4952c6765614d629fc9be85dd4baf67 2023-04-10 02:54:52,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/26e508a7ec2a4823abd248d565e5eaaa to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/26e508a7ec2a4823abd248d565e5eaaa 2023-04-10 02:54:52,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/51a235e84cef45449ae69d492fbdb4dd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/51a235e84cef45449ae69d492fbdb4dd 2023-04-10 02:54:52,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/98d5ed57ccee439db0667428fe6c9665 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/98d5ed57ccee439db0667428fe6c9665 2023-04-10 02:54:52,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b180dd1bf8474d4db026fc4c1e56bf0e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b180dd1bf8474d4db026fc4c1e56bf0e 2023-04-10 02:54:52,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/73ada6a07dcb49f1a6b55ed7da880bc7 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/73ada6a07dcb49f1a6b55ed7da880bc7 2023-04-10 02:54:52,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b002b2e7673b4245bc0f5886ee2038c1 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b002b2e7673b4245bc0f5886ee2038c1 2023-04-10 02:54:52,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/ee6db7ce430b4a738df078ac0cc3e647 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/ee6db7ce430b4a738df078ac0cc3e647 2023-04-10 02:54:52,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b2a9ba1195944567ad0563b712e064eb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b2a9ba1195944567ad0563b712e064eb 2023-04-10 02:54:52,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/34802c322c824c9b8890ccd396140d09 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/34802c322c824c9b8890ccd396140d09 2023-04-10 02:54:52,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/da0c282a0c52472f94357fe89467e9cc to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/da0c282a0c52472f94357fe89467e9cc 2023-04-10 02:54:52,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/a44cccd818c94493af87e52eb556b234 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/a44cccd818c94493af87e52eb556b234 2023-04-10 02:54:52,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/9e516e3a294b46cc8fbb9c92f733406f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/9e516e3a294b46cc8fbb9c92f733406f 2023-04-10 02:54:52,652 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/2124b1fd6cac4205a7687a0f14b0a81a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/2124b1fd6cac4205a7687a0f14b0a81a 2023-04-10 02:54:52,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/05bfb749a70240569a37492d5e60d008 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/05bfb749a70240569a37492d5e60d008 2023-04-10 02:54:52,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b7f2cf700fce4c3bb25e173b5849a8bf to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/b7f2cf700fce4c3bb25e173b5849a8bf 2023-04-10 02:54:52,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/665f18ea269940b48749adc683d926cd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/665f18ea269940b48749adc683d926cd 2023-04-10 02:54:52,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/1459e19a118e4205bb481e758627902b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/1459e19a118e4205bb481e758627902b 2023-04-10 02:54:52,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/6d4843b28c324e3b8910ecc1dc20e351 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/6d4843b28c324e3b8910ecc1dc20e351 2023-04-10 02:54:52,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/6df9067e043e49a5888b5a77b27fbb1e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/eaa106e1299a46d1966ab13df029cb28, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/058bda01b35741dfa2e0c3e22c71533b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/985ab9a5fb2e4bf886161c356e519108, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/bc6c1919b19943d98c7ead72f4618e70, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d73ea016e6ad4e26949e03c161dacc1b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/c428dc41a1834d909a831b0c66b481a0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/1dcb489f700643d8bea75bae38a9ccb3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5e9b20c088af49459195955a55230219, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/87cec3859eb74747a6b6bef9e8694f39, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/8f6ff9a8bf5a42e1b976e0a2688c0aa2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/115c5078929646e9b021db28cfbd47e8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d41c7ef0236f4b218f5e210e336d7214, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5e9fce30affb45dab1ff7627bc2cc85b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5aca45fb63ab47c3b6b56656b3001417, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d056f82b3e7143bf9e8b4e77dace8fcf, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/025003e13fe24b22a946b475dcfb4d23, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/2fc175114a2649c882f141f0ffdb5ab2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d64513d479ea4e62b712eae537fa093a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/7e3e82b2c2be4bdbbf97b61a4f6e7836, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/75eab5e2c488439088b0b221be88a022] to archive 2023-04-10 02:54:52,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:54:52,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/6df9067e043e49a5888b5a77b27fbb1e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/6df9067e043e49a5888b5a77b27fbb1e 2023-04-10 02:54:52,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/eaa106e1299a46d1966ab13df029cb28 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/eaa106e1299a46d1966ab13df029cb28 2023-04-10 02:54:52,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/058bda01b35741dfa2e0c3e22c71533b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/058bda01b35741dfa2e0c3e22c71533b 2023-04-10 02:54:52,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/985ab9a5fb2e4bf886161c356e519108 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/985ab9a5fb2e4bf886161c356e519108 2023-04-10 02:54:52,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/bc6c1919b19943d98c7ead72f4618e70 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/bc6c1919b19943d98c7ead72f4618e70 2023-04-10 02:54:52,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d73ea016e6ad4e26949e03c161dacc1b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d73ea016e6ad4e26949e03c161dacc1b 2023-04-10 02:54:52,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/c428dc41a1834d909a831b0c66b481a0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/c428dc41a1834d909a831b0c66b481a0 2023-04-10 02:54:52,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/1dcb489f700643d8bea75bae38a9ccb3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/1dcb489f700643d8bea75bae38a9ccb3 2023-04-10 02:54:52,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5e9b20c088af49459195955a55230219 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5e9b20c088af49459195955a55230219 2023-04-10 02:54:52,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/87cec3859eb74747a6b6bef9e8694f39 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/87cec3859eb74747a6b6bef9e8694f39 2023-04-10 02:54:52,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/8f6ff9a8bf5a42e1b976e0a2688c0aa2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/8f6ff9a8bf5a42e1b976e0a2688c0aa2 2023-04-10 02:54:52,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/115c5078929646e9b021db28cfbd47e8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/115c5078929646e9b021db28cfbd47e8 2023-04-10 02:54:52,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d41c7ef0236f4b218f5e210e336d7214 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d41c7ef0236f4b218f5e210e336d7214 2023-04-10 02:54:52,680 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5e9fce30affb45dab1ff7627bc2cc85b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5e9fce30affb45dab1ff7627bc2cc85b 2023-04-10 02:54:52,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5aca45fb63ab47c3b6b56656b3001417 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/5aca45fb63ab47c3b6b56656b3001417 2023-04-10 02:54:52,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d056f82b3e7143bf9e8b4e77dace8fcf to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d056f82b3e7143bf9e8b4e77dace8fcf 2023-04-10 02:54:52,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/025003e13fe24b22a946b475dcfb4d23 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/025003e13fe24b22a946b475dcfb4d23 2023-04-10 02:54:52,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/2fc175114a2649c882f141f0ffdb5ab2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/2fc175114a2649c882f141f0ffdb5ab2 2023-04-10 02:54:52,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d64513d479ea4e62b712eae537fa093a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d64513d479ea4e62b712eae537fa093a 2023-04-10 02:54:52,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/7e3e82b2c2be4bdbbf97b61a4f6e7836 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/7e3e82b2c2be4bdbbf97b61a4f6e7836 2023-04-10 02:54:52,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/75eab5e2c488439088b0b221be88a022 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/75eab5e2c488439088b0b221be88a022 2023-04-10 02:54:52,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/259c48e4c3a649d1b9aa2da203082bee, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/5cdaa54fa5c94479a4095b5a2416e8c6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/20ba4388b40f46509623cc0a316ca48f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/3595ff077d144d0a9108495086d0cad0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/c5c96b26056b4130a418f78e2d4db46b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/882226a8ad454e18b5e2598817da510d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/d71a3d8b28ca45658a076a134c4f9d63, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/7ec80126129845f0973c2423455f7a0e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/a0fbd28555d9433ab4e9d576076e439f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/19a33b25e5694f87ba6eb98d60c8cb0c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/68e6d63f18c346bf9bc6610360332c2f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/bcdfd786491f44bfadaa2f211a79b67d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/21519a1cb100437380bbc32a707ca2a2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/38edeac5f2dd41d09cefd9caf08b15f5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/6be3456d2b7542d29253bb8e96f877c7, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/55f7f1cc13b44a7b824acc35d4680296, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/57eb9cfbe2104822ab24995b451afb90, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/1e4bc142ae444301bca8e3a30435c885, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/225a3a7b35284dcbae1071191159ff03, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/df87cff819904432bfa05e49ad2863e5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/8f1d16836dce4ebb90c25dde6fb72107] to archive 2023-04-10 02:54:52,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:54:52,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/259c48e4c3a649d1b9aa2da203082bee to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/259c48e4c3a649d1b9aa2da203082bee 2023-04-10 02:54:52,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/5cdaa54fa5c94479a4095b5a2416e8c6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/5cdaa54fa5c94479a4095b5a2416e8c6 2023-04-10 02:54:52,692 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/20ba4388b40f46509623cc0a316ca48f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/20ba4388b40f46509623cc0a316ca48f 2023-04-10 02:54:52,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/3595ff077d144d0a9108495086d0cad0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/3595ff077d144d0a9108495086d0cad0 2023-04-10 02:54:52,694 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/c5c96b26056b4130a418f78e2d4db46b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/c5c96b26056b4130a418f78e2d4db46b 2023-04-10 02:54:52,695 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/882226a8ad454e18b5e2598817da510d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/882226a8ad454e18b5e2598817da510d 2023-04-10 02:54:52,695 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/d71a3d8b28ca45658a076a134c4f9d63 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/d71a3d8b28ca45658a076a134c4f9d63 2023-04-10 02:54:52,696 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/7ec80126129845f0973c2423455f7a0e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/7ec80126129845f0973c2423455f7a0e 2023-04-10 02:54:52,697 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/a0fbd28555d9433ab4e9d576076e439f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/a0fbd28555d9433ab4e9d576076e439f 2023-04-10 02:54:52,698 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/19a33b25e5694f87ba6eb98d60c8cb0c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/19a33b25e5694f87ba6eb98d60c8cb0c 2023-04-10 02:54:52,699 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/68e6d63f18c346bf9bc6610360332c2f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/68e6d63f18c346bf9bc6610360332c2f 2023-04-10 02:54:52,700 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/bcdfd786491f44bfadaa2f211a79b67d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/bcdfd786491f44bfadaa2f211a79b67d 2023-04-10 02:54:52,701 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/21519a1cb100437380bbc32a707ca2a2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/21519a1cb100437380bbc32a707ca2a2 2023-04-10 02:54:52,702 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/38edeac5f2dd41d09cefd9caf08b15f5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/38edeac5f2dd41d09cefd9caf08b15f5 2023-04-10 02:54:52,703 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/6be3456d2b7542d29253bb8e96f877c7 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/6be3456d2b7542d29253bb8e96f877c7 2023-04-10 02:54:52,704 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/55f7f1cc13b44a7b824acc35d4680296 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/55f7f1cc13b44a7b824acc35d4680296 2023-04-10 02:54:52,706 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/57eb9cfbe2104822ab24995b451afb90 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/57eb9cfbe2104822ab24995b451afb90 2023-04-10 02:54:52,707 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/1e4bc142ae444301bca8e3a30435c885 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/1e4bc142ae444301bca8e3a30435c885 2023-04-10 02:54:52,708 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/225a3a7b35284dcbae1071191159ff03 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/225a3a7b35284dcbae1071191159ff03 2023-04-10 02:54:52,709 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/df87cff819904432bfa05e49ad2863e5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/df87cff819904432bfa05e49ad2863e5 2023-04-10 02:54:52,711 DEBUG [StoreCloser-TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/8f1d16836dce4ebb90c25dde6fb72107 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/8f1d16836dce4ebb90c25dde6fb72107 2023-04-10 02:54:52,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/recovered.edits/379.seqid, newMaxSeqId=379, maxSeqId=4 2023-04-10 02:54:52,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc. 2023-04-10 02:54:52,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a944754ff766fcb82f3c238c462e40cc: 2023-04-10 02:54:52,722 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:52,722 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=a944754ff766fcb82f3c238c462e40cc, regionState=CLOSED 2023-04-10 02:54:52,723 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095292722"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095292722"}]},"ts":"1681095292722"} 2023-04-10 02:54:52,725 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-10 02:54:52,725 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure a944754ff766fcb82f3c238c462e40cc, server=jenkins-hbase4.apache.org,38321,1681095222501 in 1.9150 sec 2023-04-10 02:54:52,726 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-10 02:54:52,726 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a944754ff766fcb82f3c238c462e40cc, UNASSIGN in 1.9200 sec 2023-04-10 02:54:52,727 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095292727"}]},"ts":"1681095292727"} 2023-04-10 02:54:52,728 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-10 02:54:52,729 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-10 02:54:52,731 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9320 sec 2023-04-10 02:54:52,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-10 02:54:52,905 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-10 02:54:52,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-10 02:54:52,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:52,908 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:52,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-10 02:54:52,909 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:52,914 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:52,916 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/recovered.edits] 2023-04-10 02:54:52,919 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/21cd005c36a24418bed04765994b8032 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/21cd005c36a24418bed04765994b8032 2023-04-10 02:54:52,921 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/4f3511ec093346e1a879797f1f5de624 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/4f3511ec093346e1a879797f1f5de624 2023-04-10 02:54:52,922 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/d70ada5dd9bb488ab32d81bc26ea694c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/d70ada5dd9bb488ab32d81bc26ea694c 2023-04-10 02:54:52,923 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/e1e11a7743a14458b314da05662953ca to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/A/e1e11a7743a14458b314da05662953ca 2023-04-10 02:54:52,925 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/001d2ed1fec94c2bb3cc0bd386e40ecc to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/001d2ed1fec94c2bb3cc0bd386e40ecc 2023-04-10 02:54:52,926 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/8fc1584ea3414d3c852c3f37d34360cc to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/8fc1584ea3414d3c852c3f37d34360cc 2023-04-10 02:54:52,928 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d93bcd5394c1442f8346256a8a5ed8f7 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/d93bcd5394c1442f8346256a8a5ed8f7 2023-04-10 02:54:52,929 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/e6e2a09449da41a3acaa789ada9c12a5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/B/e6e2a09449da41a3acaa789ada9c12a5 2023-04-10 02:54:52,931 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/79f3481d13674d98b2f5c01137239499 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/79f3481d13674d98b2f5c01137239499 2023-04-10 02:54:52,932 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/98ca06e4a1034b4491b087222c0d9bb4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/98ca06e4a1034b4491b087222c0d9bb4 2023-04-10 02:54:52,933 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/b9e0a9453fa24d499b37be4c575d73a4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/b9e0a9453fa24d499b37be4c575d73a4 2023-04-10 02:54:52,934 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/c52d4710e73b4cb99679f0fe147c5a42 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/C/c52d4710e73b4cb99679f0fe147c5a42 2023-04-10 02:54:52,936 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/recovered.edits/379.seqid to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc/recovered.edits/379.seqid 2023-04-10 02:54:52,936 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/a944754ff766fcb82f3c238c462e40cc 2023-04-10 02:54:52,936 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-10 02:54:52,937 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-10 02:54:52,937 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-10 02:54:52,940 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410107dfd41acb24797b23daf99c31817dd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410107dfd41acb24797b23daf99c31817dd 2023-04-10 02:54:52,941 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041023d067f07a2e4bb693644e7e5a81b0fa to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041023d067f07a2e4bb693644e7e5a81b0fa 2023-04-10 02:54:52,942 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041025a31c6f36a04c6d95f3e526403d0b08 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041025a31c6f36a04c6d95f3e526403d0b08 2023-04-10 02:54:52,943 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041033dfc16ffb124aa3a4efae37be7990b4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041033dfc16ffb124aa3a4efae37be7990b4 2023-04-10 02:54:52,944 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304103c2e70a862ea42ad9ea1fc4049826fbb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304103c2e70a862ea42ad9ea1fc4049826fbb 2023-04-10 02:54:52,945 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041052c19ca38d1d4a37b878b37c69153216 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041052c19ca38d1d4a37b878b37c69153216 2023-04-10 02:54:52,946 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410577e1e32583146cf8593b1bf726cb28a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410577e1e32583146cf8593b1bf726cb28a 2023-04-10 02:54:52,947 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041069aff50aa3ef432cb60bd72b70fdeb9c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041069aff50aa3ef432cb60bd72b70fdeb9c 2023-04-10 02:54:52,948 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304107aced8f9be614bd6a2639cdf7b9e6947 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304107aced8f9be614bd6a2639cdf7b9e6947 2023-04-10 02:54:52,949 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304107b6ce678278844a4ad7ccdfcaf7f687e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304107b6ce678278844a4ad7ccdfcaf7f687e 2023-04-10 02:54:52,950 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041095a58f20ab9d40ddb4d65ca3d7aa0eaf to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041095a58f20ab9d40ddb4d65ca3d7aa0eaf 2023-04-10 02:54:52,951 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304109c1a16af986e4b95a9082e751af779f7 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304109c1a16af986e4b95a9082e751af779f7 2023-04-10 02:54:52,951 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a35b83d161bc4568b7af35181f2de4a5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a35b83d161bc4568b7af35181f2de4a5 2023-04-10 02:54:52,952 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a568ad8f84864746b2a1fb1ba12fdd79 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a568ad8f84864746b2a1fb1ba12fdd79 2023-04-10 02:54:52,953 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410c291308d03b640278e00c01a358e2a6f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410c291308d03b640278e00c01a358e2a6f 2023-04-10 02:54:52,954 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410d2274c0ed5284c50a9883670aafbdfc9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410d2274c0ed5284c50a9883670aafbdfc9 2023-04-10 02:54:52,956 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410e44b588ad30b4554a4c6dc0a3a0d88cb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410e44b588ad30b4554a4c6dc0a3a0d88cb 2023-04-10 02:54:52,957 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410e735f45560094bc48bb023b677b2a967 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410e735f45560094bc48bb023b677b2a967 2023-04-10 02:54:52,958 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410e836275427084327bd7fac5ea785aa08 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410e836275427084327bd7fac5ea785aa08 2023-04-10 02:54:52,959 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fab5de39edaa44eb91f9508448d18e33 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fab5de39edaa44eb91f9508448d18e33 2023-04-10 02:54:52,959 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-10 02:54:52,961 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:52,964 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-10 02:54:52,966 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-10 02:54:52,967 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:52,967 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-10 02:54:52,967 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681095292967"}]},"ts":"9223372036854775807"} 2023-04-10 02:54:52,969 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-10 02:54:52,969 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a944754ff766fcb82f3c238c462e40cc, NAME => 'TestAcidGuarantees,,1681095261160.a944754ff766fcb82f3c238c462e40cc.', STARTKEY => '', ENDKEY => ''}] 2023-04-10 02:54:52,969 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-10 02:54:52,969 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681095292969"}]},"ts":"9223372036854775807"} 2023-04-10 02:54:52,970 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-10 02:54:52,973 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:52,974 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 67 msec 2023-04-10 02:54:53,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-10 02:54:53,009 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-10 02:54:53,022 INFO [Listener at localhost/44533] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 231), OpenFileDescriptor=527 (was 542), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=392 (was 334) - SystemLoadAverage LEAK? -, ProcessCount=184 (was 184), AvailableMemoryMB=1538 (was 1734) 2023-04-10 02:54:53,033 INFO [Listener at localhost/44533] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=527, MaxFileDescriptor=60000, SystemLoadAverage=392, ProcessCount=184, AvailableMemoryMB=1538 2023-04-10 02:54:53,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-10 02:54:53,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-10 02:54:53,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-10 02:54:53,038 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-10 02:54:53,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-10 02:54:53,038 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-10 02:54:53,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-10 02:54:53,039 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:53,040 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947 empty. 2023-04-10 02:54:53,040 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:53,040 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-10 02:54:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-10 02:54:53,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-10 02:54:53,448 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-10 02:54:53,450 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => de5c1c1f03450ada9cd6207f5cfca947, NAME => 'TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp 2023-04-10 02:54:53,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-10 02:54:53,855 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:54:53,855 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing de5c1c1f03450ada9cd6207f5cfca947, disabling compactions & flushes 2023-04-10 02:54:53,855 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:53,855 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:53,855 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. after waiting 0 ms 2023-04-10 02:54:53,855 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:53,855 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:53,855 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:54:53,858 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-10 02:54:53,858 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095293858"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095293858"}]},"ts":"1681095293858"} 2023-04-10 02:54:53,859 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-10 02:54:53,860 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-10 02:54:53,860 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095293860"}]},"ts":"1681095293860"} 2023-04-10 02:54:53,861 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-10 02:54:53,863 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=de5c1c1f03450ada9cd6207f5cfca947, ASSIGN}] 2023-04-10 02:54:53,865 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=de5c1c1f03450ada9cd6207f5cfca947, ASSIGN 2023-04-10 02:54:53,865 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=de5c1c1f03450ada9cd6207f5cfca947, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38321,1681095222501; forceNewPlan=false, retain=false 2023-04-10 02:54:54,016 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=de5c1c1f03450ada9cd6207f5cfca947, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:54,017 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095294016"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095294016"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095294016"}]},"ts":"1681095294016"} 2023-04-10 02:54:54,018 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:54:54,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-10 02:54:54,173 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:54,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => de5c1c1f03450ada9cd6207f5cfca947, NAME => 'TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.', STARTKEY => '', ENDKEY => ''} 2023-04-10 02:54:54,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:54,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:54:54,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:54,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:54,176 INFO [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:54,178 DEBUG [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A 2023-04-10 02:54:54,178 DEBUG [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A 2023-04-10 02:54:54,179 INFO [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:54:54,179 INFO [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region de5c1c1f03450ada9cd6207f5cfca947 columnFamilyName A 2023-04-10 02:54:54,179 INFO [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] regionserver.HStore(310): Store=de5c1c1f03450ada9cd6207f5cfca947/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:54:54,180 INFO [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:54,181 DEBUG [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B 2023-04-10 02:54:54,181 DEBUG [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B 2023-04-10 02:54:54,181 INFO [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:54:54,181 INFO [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region de5c1c1f03450ada9cd6207f5cfca947 columnFamilyName B 2023-04-10 02:54:54,182 INFO [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] regionserver.HStore(310): Store=de5c1c1f03450ada9cd6207f5cfca947/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:54:54,182 INFO [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:54,183 DEBUG [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C 2023-04-10 02:54:54,183 DEBUG [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C 2023-04-10 02:54:54,183 INFO [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:54:54,184 INFO [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region de5c1c1f03450ada9cd6207f5cfca947 columnFamilyName C 2023-04-10 02:54:54,184 INFO [StoreOpener-de5c1c1f03450ada9cd6207f5cfca947-1] regionserver.HStore(310): Store=de5c1c1f03450ada9cd6207f5cfca947/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:54:54,184 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:54,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:54,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:54,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-10 02:54:54,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:54,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-10 02:54:54,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened de5c1c1f03450ada9cd6207f5cfca947; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61961116, jitterRate=-0.07670742273330688}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-10 02:54:54,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:54:54,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., pid=30, masterSystemTime=1681095294169 2023-04-10 02:54:54,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:54,195 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:54,196 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=de5c1c1f03450ada9cd6207f5cfca947, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:54,196 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095294196"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681095294196"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681095294196"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681095294196"}]},"ts":"1681095294196"} 2023-04-10 02:54:54,199 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-10 02:54:54,199 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 in 179 msec 2023-04-10 02:54:54,201 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-10 02:54:54,201 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=de5c1c1f03450ada9cd6207f5cfca947, ASSIGN in 336 msec 2023-04-10 02:54:54,204 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-10 02:54:54,204 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095294204"}]},"ts":"1681095294204"} 2023-04-10 02:54:54,205 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-10 02:54:54,208 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-10 02:54:54,210 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1720 sec 2023-04-10 02:54:55,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-10 02:54:55,143 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-10 02:54:55,144 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x112921f2 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:55,149 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22b4eae5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:55,150 DEBUG [Listener at localhost/44533] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:55,151 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:55,153 DEBUG [Listener at localhost/44533] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-10 02:54:55,154 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41172, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-10 02:54:55,155 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x7ed1dde9 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:55,159 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@262f4570, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:55,159 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x70043cf0 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:55,163 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ca898c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:55,163 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:55,171 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:55,172 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:55,175 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:55,176 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x330b77da to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:55,179 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a24d617, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:55,179 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x104dbf4e to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:55,185 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ce5573e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:55,185 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x753156b2 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:55,194 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@87b3095, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:55,194 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:55,199 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:55,199 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:55,206 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:55,206 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x5c43dddd to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:54:55,210 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbaba8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:54:55,218 DEBUG [hconnection-0x2b70e3e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:55,218 DEBUG [hconnection-0xf85e176-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:55,218 DEBUG [hconnection-0xc416527-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:55,219 DEBUG [hconnection-0x52ebddbf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:55,219 DEBUG [hconnection-0x63d876cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:55,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:55,220 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40360, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:55,221 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40376, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:55,220 DEBUG [hconnection-0x7c38e080-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:55,221 DEBUG [hconnection-0x74ae6e64-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:55,221 DEBUG [hconnection-0x7f6ebbca-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:55,222 DEBUG [hconnection-0x4d950b0b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:55,222 DEBUG [hconnection-0x2b10ee1d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:54:55,223 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40362, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:55,224 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40390, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:55,224 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40400, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:55,224 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40414, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:55,226 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40426, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:55,228 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40424, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:55,233 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40418, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:55,234 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40430, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:54:55,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:55,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:55,235 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:55,235 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:55,235 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:55,235 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:55,236 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,236 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,238 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,238 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:55,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:55,240 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:55,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:55,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:55,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:55,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:54:55,244 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:55,245 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:55,245 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:55,245 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:54:55,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:55,246 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:54:55,246 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:55,246 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:54:55,246 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:55,253 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:55,253 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:55,253 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:55,253 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:55,253 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:55,253 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:55,253 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:55,253 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:55,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:55,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:55,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,255 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:55,256 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,256 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:55,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:55,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3abe1924[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:55,258 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:55,258 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:55,260 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,260 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:55,261 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:55,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:55,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:55,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:55,264 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:55,264 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:55,264 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:55,264 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:55,264 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:55,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,267 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:55,267 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:55,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:55,267 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,269 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095355266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095355270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095355270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095355271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,274 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:55,274 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,274 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,274 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:55,274 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:55,274 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:55,275 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,275 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095355271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:55,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:55,275 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:55,275 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,275 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:55,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:55,275 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:55,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095355372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,375 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:55,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:55,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095355374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095355375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095355375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:55,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095355376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:55,379 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:55,380 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:55,380 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:55,380 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:55,380 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,380 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:55,381 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:55,382 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,382 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:55,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:55,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:55,388 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:55,388 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:55,388 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:55,392 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:55,392 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:55,392 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:55,392 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:55,395 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,395 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:55,395 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,395 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:55,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:55,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72f8b43a[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:55,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:55,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,398 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:55,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:55,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:55,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:55,400 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,400 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:55,400 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:55,400 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:55,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:55,402 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:55,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:55,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:55,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:55,403 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:55,405 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:55,405 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,405 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:55,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:55,405 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,407 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,411 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:55,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:55,411 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:55,411 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,411 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,411 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,412 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,412 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,412 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,412 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:55,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:55,412 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:55,412 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:55,412 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:55,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:55,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:55,512 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:55,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:55,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:55,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:55,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:55,516 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:55,516 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:55,516 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:55,516 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:55,517 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,517 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,518 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:55,518 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:55,518 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,519 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:55,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:55,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:55,522 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:55,522 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:55,522 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:55,522 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,523 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:55,523 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:55,524 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,524 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:55,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:55,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@769d382f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:55,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:55,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,527 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:55,528 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,528 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:55,531 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,531 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:55,531 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:55,531 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:55,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:55,533 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:55,533 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:55,533 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:55,533 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:55,533 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:55,535 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,535 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,535 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:55,536 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,537 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,539 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,539 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:55,539 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:55,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:55,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:55,539 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:55,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:55,540 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,540 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,540 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,540 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,540 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,540 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:55,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:55,540 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:55,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:55,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095355575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095355577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095355578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095355578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095355580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,640 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:55,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:55,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:55,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:55,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:55,644 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:55,644 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:55,644 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:55,644 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:55,644 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,644 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,646 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,646 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:55,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:55,646 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,646 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:55,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:55,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:55,647 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:55,647 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:55,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:55,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:55,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:55,648 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:55,648 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:55,648 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:55,648 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:55,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:55,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:55,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:55,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:55,651 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,651 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:55,651 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,651 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:55,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:55,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@366728f4[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:55,653 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:55,653 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,654 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:55,655 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:55,655 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:55,655 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,655 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:55,656 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:55,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:55,659 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:55,659 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:55,659 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:55,659 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:55,659 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:55,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:55,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:55,661 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,663 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,666 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,666 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,666 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,666 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,666 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,666 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:55,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:55,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:55,666 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,666 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:55,666 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:55,666 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:55,666 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:55,667 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:55,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:55,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:55,716 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/39981446dd5d4a6c8147daf8a60356cb 2023-04-10 02:54:55,729 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/07dc44ae6e974c97827f567ff2ac555a 2023-04-10 02:54:55,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/61ec709206f14c1eaea11a6c978e7087 2023-04-10 02:54:55,744 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/39981446dd5d4a6c8147daf8a60356cb as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/39981446dd5d4a6c8147daf8a60356cb 2023-04-10 02:54:55,749 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/39981446dd5d4a6c8147daf8a60356cb, entries=150, sequenceid=13, filesize=11.6 K 2023-04-10 02:54:55,750 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/07dc44ae6e974c97827f567ff2ac555a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/07dc44ae6e974c97827f567ff2ac555a 2023-04-10 02:54:55,756 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/07dc44ae6e974c97827f567ff2ac555a, entries=150, sequenceid=13, filesize=11.6 K 2023-04-10 02:54:55,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/61ec709206f14c1eaea11a6c978e7087 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/61ec709206f14c1eaea11a6c978e7087 2023-04-10 02:54:55,767 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:55,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/61ec709206f14c1eaea11a6c978e7087, entries=150, sequenceid=13, filesize=11.6 K 2023-04-10 02:54:55,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:55,768 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for de5c1c1f03450ada9cd6207f5cfca947 in 524ms, sequenceid=13, compaction requested=false 2023-04-10 02:54:55,768 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:54:55,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:55,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:55,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:55,770 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:55,770 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:55,770 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:55,770 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:55,770 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,770 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:55,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:55,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:55,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,774 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:55,775 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,775 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:55,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:55,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:55,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:55,776 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:55,776 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:55,780 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:55,780 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:55,780 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-10 02:54:55,780 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:54:55,780 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:55,780 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:54:55,780 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:55,781 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:54:55,781 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:55,792 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/7d3887e3e1ce420dab862b9701b1159a 2023-04-10 02:54:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:55,880 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:55,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095355886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095355887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095355886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095355890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095355890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095355990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095355991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095355991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095355993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:55,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:55,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095355993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:56,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095356193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:56,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:56,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095356194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095356194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:56,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095356197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:56,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095356197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,235 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/9cd25fcb2f6645ad8a5020d65fd3e405 2023-04-10 02:54:56,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:56,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095356495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:56,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095356497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:56,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095356498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:56,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095356499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:56,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095356500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,647 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/65fa94c563004511a490a5bce9f50908 2023-04-10 02:54:56,652 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/7d3887e3e1ce420dab862b9701b1159a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/7d3887e3e1ce420dab862b9701b1159a 2023-04-10 02:54:56,656 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/7d3887e3e1ce420dab862b9701b1159a, entries=150, sequenceid=33, filesize=11.6 K 2023-04-10 02:54:56,657 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/9cd25fcb2f6645ad8a5020d65fd3e405 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9cd25fcb2f6645ad8a5020d65fd3e405 2023-04-10 02:54:56,663 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9cd25fcb2f6645ad8a5020d65fd3e405, entries=150, sequenceid=33, filesize=11.6 K 2023-04-10 02:54:56,664 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/65fa94c563004511a490a5bce9f50908 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/65fa94c563004511a490a5bce9f50908 2023-04-10 02:54:56,667 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/65fa94c563004511a490a5bce9f50908, entries=150, sequenceid=33, filesize=11.6 K 2023-04-10 02:54:56,668 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for de5c1c1f03450ada9cd6207f5cfca947 in 888ms, sequenceid=33, compaction requested=false 2023-04-10 02:54:56,668 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:54:56,669 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:56,670 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:56,670 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:56,670 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,670 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:56,670 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:56,673 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,673 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:56,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:56,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cda7025[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:56,676 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:56,676 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,678 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,678 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:56,681 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:56,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:56,684 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:56,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:56,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:56,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:56,684 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:56,686 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,687 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:56,687 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:56,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:56,687 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,689 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,693 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:56,693 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:56,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:56,693 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,693 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:56,693 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:56,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:56,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:56,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:56,693 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:56,693 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:56,693 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:56,693 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,693 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,694 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,694 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,793 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:56,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:56,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:56,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:56,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:56,807 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:56,807 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:56,807 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:56,808 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:56,808 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,808 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:56,809 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,809 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:56,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:56,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:56,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:56,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:56,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:56,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:56,812 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:56,812 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:56,812 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:56,824 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:56,824 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:56,832 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:56,832 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:56,833 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-10 02:54:56,833 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:54:56,833 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:56,833 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:54:56,833 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:56,833 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:54:56,833 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:56,860 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/41898603d8f84aba8079ca83c3a8cc8b 2023-04-10 02:54:56,883 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/2e5276274f4344ff8d26f36cbf257613 2023-04-10 02:54:56,896 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/56548abb001a4042a81b2fc956b0050a 2023-04-10 02:54:56,902 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/41898603d8f84aba8079ca83c3a8cc8b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/41898603d8f84aba8079ca83c3a8cc8b 2023-04-10 02:54:56,905 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/41898603d8f84aba8079ca83c3a8cc8b, entries=150, sequenceid=46, filesize=11.6 K 2023-04-10 02:54:56,906 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/2e5276274f4344ff8d26f36cbf257613 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/2e5276274f4344ff8d26f36cbf257613 2023-04-10 02:54:56,927 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/2e5276274f4344ff8d26f36cbf257613, entries=150, sequenceid=46, filesize=11.6 K 2023-04-10 02:54:56,931 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/56548abb001a4042a81b2fc956b0050a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/56548abb001a4042a81b2fc956b0050a 2023-04-10 02:54:56,939 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/56548abb001a4042a81b2fc956b0050a, entries=150, sequenceid=46, filesize=11.6 K 2023-04-10 02:54:56,939 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for de5c1c1f03450ada9cd6207f5cfca947 in 106ms, sequenceid=46, compaction requested=true 2023-04-10 02:54:56,939 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:54:56,940 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:56,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:56,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:56,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:56,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:56,941 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:56,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:56,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:56,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:56,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:56,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:56,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b3442e0[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:56,945 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:56,945 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:56,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:56,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:56,948 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,948 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:56,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:56,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:56,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:56,950 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:56,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:56,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:56,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:56,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:56,954 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:56,954 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:56,954 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:56,954 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:56,954 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:56,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:56,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:56,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:56,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:56,965 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:56,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:56,966 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:56,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:56,969 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:56,969 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:56,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:56,969 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:56,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:56,969 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:56,969 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:56,970 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:56,970 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:56,970 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,970 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:56,970 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:56,970 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:56,970 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:56,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:56,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:57,015 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:54:57,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:54:57,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:57,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:54:57,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:57,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:54:57,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:57,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:57,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681095297021, max timestamp=1681095297037Num uniques 859; 2023-04-10 02:54:57,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:54:57,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681095297021, max timestamp=1681095297037Num uniques 859; 2023-04-10 02:54:57,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:54:57,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095297021, max timestamp=1681095297037Num uniques 859; 2023-04-10 02:54:57,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:54:57,065 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=56 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/9e10cce422ba4712bdde1e0700166643 2023-04-10 02:54:57,070 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:57,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:57,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:57,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095357071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:57,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095357074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095357074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:57,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095357075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095357074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,080 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:57,080 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,080 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:57,080 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:57,081 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,081 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,082 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,083 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,083 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,083 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:57,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:57,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:57,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:57,088 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:57,088 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,088 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:57,096 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:57,096 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:57,096 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,096 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:57,096 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:57,096 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,096 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:57,096 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:57,097 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,097 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:57,097 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:57,098 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,098 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,098 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,099 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:57,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:57,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547747e9[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:57,101 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:57,102 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,103 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,103 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:57,103 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:57,103 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:57,103 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,103 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:57,105 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:57,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:57,109 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:57,109 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:57,109 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:57,109 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:57,109 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:57,113 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,113 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,113 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,114 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,115 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,118 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:57,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:57,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,119 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:57,119 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:57,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:57,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095357178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095357181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095357181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095357181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095357181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,219 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:57,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:57,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:57,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:57,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:57,222 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:57,222 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,223 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:57,223 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:57,223 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,223 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,224 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,225 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,225 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:57,228 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:57,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:57,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:57,238 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,238 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:57,238 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:57,238 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:57,240 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,240 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,240 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,240 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:57,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bd8f51b[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:57,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:57,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,247 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:57,249 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:57,249 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,249 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:57,249 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:57,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:57,252 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:57,252 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:57,252 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:57,252 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:57,252 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:57,253 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,253 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,253 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,254 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,256 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,260 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,260 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,260 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,260 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:57,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:57,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:57,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,260 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,260 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,260 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,260 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,260 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:57,260 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,260 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:57,261 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,361 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:57,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:57,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:57,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:57,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:57,364 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:57,364 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,364 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:57,364 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:57,364 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,364 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,365 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,366 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,366 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,366 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:57,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:57,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:57,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:57,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:57,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:57,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:57,369 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,369 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:57,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@742cfd1d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:57,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:57,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,372 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,372 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:57,372 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:57,372 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:57,373 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,373 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:57,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,375 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:57,375 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,375 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:57,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:57,377 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:57,377 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:57,377 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:57,377 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:57,377 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:57,379 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,379 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,379 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,380 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,381 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,385 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,385 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:57,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:57,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095357383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,385 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,385 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,385 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,385 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,385 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,385 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,385 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:57,385 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,385 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:57,386 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:57,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095357384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095357384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095357384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095357385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,486 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:57,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:57,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:57,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:57,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:57,489 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:57,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:57,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:57,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,490 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,490 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=56 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/e7421710c0b9455388a655b91ba68c9f 2023-04-10 02:54:57,491 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:57,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:57,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:57,493 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,494 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:57,494 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:57,494 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:57,497 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,497 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,498 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,498 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d66823a[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:57,500 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:57,500 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,501 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,501 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:57,501 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:57,501 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:57,501 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,501 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:57,503 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:57,503 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,503 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,503 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:57,505 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:57,505 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:57,505 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:57,505 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:57,505 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:57,507 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,507 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,507 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,507 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,508 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,512 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,512 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,512 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:57,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:57,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,512 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,512 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,512 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,512 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,512 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:57,512 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,512 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,512 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:57,512 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,612 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:57,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:57,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:57,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:57,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:57,616 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:57,616 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,616 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:57,616 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:57,616 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,616 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,617 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,617 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,618 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,618 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:57,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:57,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:57,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:57,619 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,619 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:57,619 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:57,619 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,620 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:57,620 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:57,621 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,621 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,621 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,621 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:57,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7340d670[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:57,623 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,624 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,624 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:57,624 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:57,624 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:57,624 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,624 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:57,629 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:57,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:57,631 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:57,631 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:57,631 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:57,631 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:57,631 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:57,635 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,635 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,635 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,636 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,637 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,639 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,639 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,639 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:57,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,639 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,639 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,640 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:57,640 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:57,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095357686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095357687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095357688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095357689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:57,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095357689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,740 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:57,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:57,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:57,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:57,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:57,746 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:57,746 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,746 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:57,746 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:57,747 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,747 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,749 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,749 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,750 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,750 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:57,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:57,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:57,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:57,753 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:57,753 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:57,753 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,753 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:57,754 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:57,754 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:57,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:57,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:57,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:57,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,755 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,755 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,755 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:57,757 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:57,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fac952e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:57,757 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,758 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,758 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:57,758 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,758 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:57,758 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:57,758 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:57,759 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,759 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,759 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:57,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:57,762 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:57,763 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:57,763 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:57,763 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:57,763 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:57,765 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,765 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,765 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,766 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,767 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,769 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,769 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,769 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,769 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,769 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:57,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:57,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,769 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,770 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:57,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,770 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,870 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:57,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:57,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:57,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:57,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:57,874 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:57,874 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,874 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:57,874 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:57,874 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,874 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,876 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,876 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,876 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,876 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:57,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:57,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:57,877 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,877 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:57,878 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:57,878 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:57,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:57,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:57,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:57,879 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,879 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,879 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,879 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:57,881 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:57,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cfb49d3[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:57,881 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,883 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,883 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:57,883 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:57,883 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:57,884 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,884 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:57,885 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:57,885 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,885 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:57,886 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:57,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:57,894 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:57,894 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:57,894 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:57,894 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:57,894 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:57,897 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,897 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:57,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:57,899 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,901 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:57,902 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=56 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/d5fce69634e9409f887f7720d9e626df 2023-04-10 02:54:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,903 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:57,903 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:57,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:57,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:57,903 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:57,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:57,903 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,904 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:57,904 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:57,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:57,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:57,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:57,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:57,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:57,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:57,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/9e10cce422ba4712bdde1e0700166643 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/9e10cce422ba4712bdde1e0700166643 2023-04-10 02:54:57,911 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/9e10cce422ba4712bdde1e0700166643, entries=100, sequenceid=56, filesize=9.3 K 2023-04-10 02:54:57,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/e7421710c0b9455388a655b91ba68c9f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/e7421710c0b9455388a655b91ba68c9f 2023-04-10 02:54:57,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/e7421710c0b9455388a655b91ba68c9f, entries=100, sequenceid=56, filesize=9.3 K 2023-04-10 02:54:57,918 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/d5fce69634e9409f887f7720d9e626df as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/d5fce69634e9409f887f7720d9e626df 2023-04-10 02:54:57,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/d5fce69634e9409f887f7720d9e626df, entries=100, sequenceid=56, filesize=9.3 K 2023-04-10 02:54:57,924 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for de5c1c1f03450ada9cd6207f5cfca947 in 909ms, sequenceid=56, compaction requested=true 2023-04-10 02:54:57,924 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:54:57,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:57,924 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:54:57,924 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:54:57,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:57,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:57,927 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:54:57,932 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:54:57,932 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/A is initiating minor compaction (all files) 2023-04-10 02:54:57,932 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/B is initiating minor compaction (all files) 2023-04-10 02:54:57,932 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/A in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,933 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/B in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,933 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/39981446dd5d4a6c8147daf8a60356cb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/7d3887e3e1ce420dab862b9701b1159a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/41898603d8f84aba8079ca83c3a8cc8b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/9e10cce422ba4712bdde1e0700166643] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=44.1 K 2023-04-10 02:54:57,933 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/07dc44ae6e974c97827f567ff2ac555a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9cd25fcb2f6645ad8a5020d65fd3e405, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/2e5276274f4344ff8d26f36cbf257613, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/e7421710c0b9455388a655b91ba68c9f] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=44.1 K 2023-04-10 02:54:57,940 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 39981446dd5d4a6c8147daf8a60356cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681095295237 2023-04-10 02:54:57,940 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 07dc44ae6e974c97827f567ff2ac555a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681095295237 2023-04-10 02:54:57,941 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 7d3887e3e1ce420dab862b9701b1159a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681095295268 2023-04-10 02:54:57,941 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 9cd25fcb2f6645ad8a5020d65fd3e405, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681095295268 2023-04-10 02:54:57,941 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 41898603d8f84aba8079ca83c3a8cc8b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681095295885 2023-04-10 02:54:57,941 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 2e5276274f4344ff8d26f36cbf257613, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681095295885 2023-04-10 02:54:57,942 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 9e10cce422ba4712bdde1e0700166643, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1681095297013 2023-04-10 02:54:57,942 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting e7421710c0b9455388a655b91ba68c9f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1681095297013 2023-04-10 02:54:57,953 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#A#compaction#213 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:57,953 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#B#compaction#214 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:57,976 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/c9cf2fea1fac4bdea1b790957c22f977 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/c9cf2fea1fac4bdea1b790957c22f977 2023-04-10 02:54:57,981 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/89d02c2a11f144d0bb9c15a63c63f140 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/89d02c2a11f144d0bb9c15a63c63f140 2023-04-10 02:54:57,982 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/B of de5c1c1f03450ada9cd6207f5cfca947 into c9cf2fea1fac4bdea1b790957c22f977(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:57,982 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:54:57,982 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/B, priority=12, startTime=1681095297924; duration=0sec 2023-04-10 02:54:57,982 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:57,982 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:54:57,983 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:54:57,984 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/C is initiating minor compaction (all files) 2023-04-10 02:54:57,984 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/C in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:57,984 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/61ec709206f14c1eaea11a6c978e7087, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/65fa94c563004511a490a5bce9f50908, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/56548abb001a4042a81b2fc956b0050a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/d5fce69634e9409f887f7720d9e626df] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=44.1 K 2023-04-10 02:54:57,985 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 61ec709206f14c1eaea11a6c978e7087, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681095295237 2023-04-10 02:54:57,985 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 65fa94c563004511a490a5bce9f50908, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681095295268 2023-04-10 02:54:57,987 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 56548abb001a4042a81b2fc956b0050a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681095295885 2023-04-10 02:54:57,987 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d5fce69634e9409f887f7720d9e626df, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1681095297013 2023-04-10 02:54:57,989 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/A of de5c1c1f03450ada9cd6207f5cfca947 into 89d02c2a11f144d0bb9c15a63c63f140(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:57,989 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:54:57,989 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/A, priority=12, startTime=1681095297924; duration=0sec 2023-04-10 02:54:57,989 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:58,000 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#C#compaction#215 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:58,004 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:58,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:58,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:58,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:58,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:58,011 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:58,011 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:58,011 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:58,011 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:58,011 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:58,012 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:58,014 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:58,014 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:58,016 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,016 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:58,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:58,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:58,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:58,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:58,017 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:58,018 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:58,018 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:58,018 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:58,018 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:58,018 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:58,018 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:58,018 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-04-10 02:54:58,018 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:54:58,019 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:58,019 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:54:58,019 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:58,020 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:54:58,020 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:54:58,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:58,192 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:58,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095358197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095358198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095358201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095358203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095358203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,255 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-10 02:54:58,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095358304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095358304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095358308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095358308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095358309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,415 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/62cc740b7e624036a0bbd9eeacf49dc4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/62cc740b7e624036a0bbd9eeacf49dc4 2023-04-10 02:54:58,420 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/C of de5c1c1f03450ada9cd6207f5cfca947 into 62cc740b7e624036a0bbd9eeacf49dc4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:54:58,420 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:54:58,420 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/C, priority=12, startTime=1681095297924; duration=0sec 2023-04-10 02:54:58,420 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:58,434 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/1960aae77cdb4c0482b6abc56cd353b2 2023-04-10 02:54:58,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095358508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095358510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095358513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095358513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095358514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095358811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095358814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095358816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095358816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:58,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095358817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:58,847 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/78545b45e3254f46b6b139577470783b 2023-04-10 02:54:59,259 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/9b8ad5e4e6ff493599c1fb91ebd25ee9 2023-04-10 02:54:59,263 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/1960aae77cdb4c0482b6abc56cd353b2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/1960aae77cdb4c0482b6abc56cd353b2 2023-04-10 02:54:59,268 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/1960aae77cdb4c0482b6abc56cd353b2, entries=150, sequenceid=94, filesize=11.6 K 2023-04-10 02:54:59,269 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/78545b45e3254f46b6b139577470783b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/78545b45e3254f46b6b139577470783b 2023-04-10 02:54:59,274 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/78545b45e3254f46b6b139577470783b, entries=150, sequenceid=94, filesize=11.6 K 2023-04-10 02:54:59,275 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/9b8ad5e4e6ff493599c1fb91ebd25ee9 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9b8ad5e4e6ff493599c1fb91ebd25ee9 2023-04-10 02:54:59,282 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9b8ad5e4e6ff493599c1fb91ebd25ee9, entries=150, sequenceid=94, filesize=11.6 K 2023-04-10 02:54:59,283 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=53.67 KB/54960 for de5c1c1f03450ada9cd6207f5cfca947 in 1264ms, sequenceid=94, compaction requested=false 2023-04-10 02:54:59,283 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:54:59,283 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:59,283 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:59,283 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:59,283 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,283 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:59,283 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,284 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:59,285 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,285 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:59,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:59,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55257324[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:59,286 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:59,286 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,287 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,288 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:59,288 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:59,288 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:59,288 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,288 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:59,289 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:59,289 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:59,289 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,289 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:59,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:59,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:59,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:59,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:59,292 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:59,292 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:59,292 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:59,292 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:59,292 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:59,294 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,294 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:59,294 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:59,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:59,295 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,297 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,301 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:59,301 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:59,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:59,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:59,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:59,301 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,301 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:59,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:59,302 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:59,302 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:59,302 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:59,302 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,302 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,302 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,302 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,302 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:59,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:59,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:54:59,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:54:59,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:59,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:54:59,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:59,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:54:59,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:59,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/64cd512df88e4a6f945e98da43c18e44 2023-04-10 02:54:59,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095359336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095359337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095359339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095359340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095359340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,402 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:59,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:59,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:59,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:59,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:59,405 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:59,405 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:59,405 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:59,405 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:59,406 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,406 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,407 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,407 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:59,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:59,407 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,407 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:59,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:59,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:59,409 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:59,409 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:59,409 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:59,409 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:59,410 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,410 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:59,410 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:59,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:59,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2023dca6[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:59,412 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:59,412 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,416 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:59,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:59,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:59,417 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,417 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:59,418 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:59,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:59,421 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:59,421 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:59,421 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:59,421 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:59,421 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:59,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:59,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,423 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:59,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:59,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:59,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:59,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,428 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:59,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:59,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:59,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:59,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,429 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:59,429 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:59,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:59,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095359441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095359441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095359444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095359444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095359444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,529 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:59,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:59,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:59,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:59,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:59,532 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:59,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:59,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:59,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:59,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,535 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:59,535 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:59,535 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,535 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:59,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:59,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:59,537 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:59,537 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:59,537 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:59,537 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:59,538 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,538 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:59,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:59,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:59,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b2d2869[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:59,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,542 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:59,542 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,542 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:59,543 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:59,546 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:59,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:59,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:59,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:59,546 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:59,548 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,548 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,548 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:59,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:59,549 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,550 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,552 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:59,552 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:59,552 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:59,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:59,552 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:59,552 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:59,552 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:59,553 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:59,553 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,553 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:59,553 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,553 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,553 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:59,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095359644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095359645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095359647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095359647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095359647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,653 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:59,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:59,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:59,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:59,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:59,656 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:59,656 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:59,656 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:59,656 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:59,657 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,657 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,658 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:59,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:59,658 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:59,658 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,658 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:59,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:59,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:59,659 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:59,660 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:59,660 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:59,660 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:59,660 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:59,660 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:59,660 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:59,660 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:59,661 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:59,661 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:54:59,661 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:54:59,661 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,661 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:54:59,661 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:54:59,662 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:59,663 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,663 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:54:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:59,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:54:59,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dbe590e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:54:59,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:54:59,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,666 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:54:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:54:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:54:59,666 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,666 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:54:59,668 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:54:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:54:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:54:59,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:54:59,670 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:54:59,670 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:54:59,670 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:54:59,670 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:54:59,670 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:54:59,672 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,672 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:54:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:54:59,672 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:59,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:54:59,673 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:54:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,674 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:54:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:54:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:59,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,680 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:54:59,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:59,680 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:54:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:59,680 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:54:59,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:54:59,680 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:54:59,680 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,680 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,680 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:54:59,680 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,680 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:54:59,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:54:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:54:59,680 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:54:59,744 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/2b14e2e5cbb748319870059b3c7cc625 2023-04-10 02:54:59,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/9a104d69a28b4bb097dfec30ac2c1b43 2023-04-10 02:54:59,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/64cd512df88e4a6f945e98da43c18e44 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/64cd512df88e4a6f945e98da43c18e44 2023-04-10 02:54:59,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/64cd512df88e4a6f945e98da43c18e44, entries=200, sequenceid=109, filesize=13.9 K 2023-04-10 02:54:59,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/2b14e2e5cbb748319870059b3c7cc625 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/2b14e2e5cbb748319870059b3c7cc625 2023-04-10 02:54:59,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/2b14e2e5cbb748319870059b3c7cc625, entries=150, sequenceid=109, filesize=11.6 K 2023-04-10 02:54:59,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/9a104d69a28b4bb097dfec30ac2c1b43 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9a104d69a28b4bb097dfec30ac2c1b43 2023-04-10 02:54:59,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9a104d69a28b4bb097dfec30ac2c1b43, entries=150, sequenceid=109, filesize=11.6 K 2023-04-10 02:54:59,780 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:54:59,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:54:59,782 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for de5c1c1f03450ada9cd6207f5cfca947 in 462ms, sequenceid=109, compaction requested=true 2023-04-10 02:54:59,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:54:59,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:54:59,782 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:59,782 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:54:59,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:59,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:54:59,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:54:59,783 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38136 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:59,783 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/A is initiating minor compaction (all files) 2023-04-10 02:54:59,783 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/A in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:59,784 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/89d02c2a11f144d0bb9c15a63c63f140, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/1960aae77cdb4c0482b6abc56cd353b2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/64cd512df88e4a6f945e98da43c18e44] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=37.2 K 2023-04-10 02:54:59,784 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 89d02c2a11f144d0bb9c15a63c63f140, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1681095295889 2023-04-10 02:54:59,784 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 1960aae77cdb4c0482b6abc56cd353b2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1681095297070 2023-04-10 02:54:59,785 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 64cd512df88e4a6f945e98da43c18e44, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681095298200 2023-04-10 02:54:59,786 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:54:59,786 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/B is initiating minor compaction (all files) 2023-04-10 02:54:59,787 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/B in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:59,787 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/c9cf2fea1fac4bdea1b790957c22f977, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/78545b45e3254f46b6b139577470783b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/2b14e2e5cbb748319870059b3c7cc625] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=35.0 K 2023-04-10 02:54:59,792 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting c9cf2fea1fac4bdea1b790957c22f977, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1681095295889 2023-04-10 02:54:59,793 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 78545b45e3254f46b6b139577470783b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1681095297070 2023-04-10 02:54:59,793 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 2b14e2e5cbb748319870059b3c7cc625, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681095298200 2023-04-10 02:54:59,802 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#B#compaction#222 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:59,802 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#A#compaction#223 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:54:59,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:54:59,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:54:59,808 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:54:59,808 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:59,808 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:54:59,808 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:54:59,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,817 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,818 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,818 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:54:59,824 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:54:59,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:54:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:54:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:54:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:54:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:54:59,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:54:59,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:54:59,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:54:59,844 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:54:59,844 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:54:59,844 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:54:59,848 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:54:59,848 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:54:59,848 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:54:59,848 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:54:59,848 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-10 02:54:59,848 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:54:59,848 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:59,848 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:54:59,849 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:59,849 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:54:59,849 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:54:59,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:54:59,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:54:59,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095359958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095359960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095359961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095359962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:54:59,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:54:59,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095359963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095360064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095360064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095360064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095360067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095360067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,239 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/b0edc22730cf4ce99fa8ad45c93cb087 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/b0edc22730cf4ce99fa8ad45c93cb087 2023-04-10 02:55:00,245 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/B of de5c1c1f03450ada9cd6207f5cfca947 into b0edc22730cf4ce99fa8ad45c93cb087(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:00,245 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:00,245 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/B, priority=13, startTime=1681095299782; duration=0sec 2023-04-10 02:55:00,245 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:00,245 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:00,246 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:00,247 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/C is initiating minor compaction (all files) 2023-04-10 02:55:00,247 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/C in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:00,247 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/62cc740b7e624036a0bbd9eeacf49dc4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9b8ad5e4e6ff493599c1fb91ebd25ee9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9a104d69a28b4bb097dfec30ac2c1b43] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=35.0 K 2023-04-10 02:55:00,247 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 62cc740b7e624036a0bbd9eeacf49dc4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1681095295889 2023-04-10 02:55:00,247 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/0a7bc07c08a047f886f1a8497f03a7ba as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/0a7bc07c08a047f886f1a8497f03a7ba 2023-04-10 02:55:00,249 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 9b8ad5e4e6ff493599c1fb91ebd25ee9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1681095297070 2023-04-10 02:55:00,250 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 9a104d69a28b4bb097dfec30ac2c1b43, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681095298200 2023-04-10 02:55:00,255 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/A of de5c1c1f03450ada9cd6207f5cfca947 into 0a7bc07c08a047f886f1a8497f03a7ba(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:00,255 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:00,255 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/A, priority=13, startTime=1681095299782; duration=0sec 2023-04-10 02:55:00,255 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:00,262 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/737aa86776b543979f9fcf4591cc03d4 2023-04-10 02:55:00,265 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#C#compaction#225 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:00,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095360268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095360269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095360270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095360271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095360272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,309 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/6515e6271a7f4b3eb3bdac25a58ea3a9 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/6515e6271a7f4b3eb3bdac25a58ea3a9 2023-04-10 02:55:00,312 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/9de45e5f092a46c1a9eed75f6e6596d8 2023-04-10 02:55:00,325 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/C of de5c1c1f03450ada9cd6207f5cfca947 into 6515e6271a7f4b3eb3bdac25a58ea3a9(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:00,325 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:00,326 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/C, priority=13, startTime=1681095299782; duration=0sec 2023-04-10 02:55:00,326 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:00,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095360575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095360575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095360576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095360576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:00,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095360579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,748 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/5f5fccf3164945d7835edf4e24607247 2023-04-10 02:55:00,754 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/737aa86776b543979f9fcf4591cc03d4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/737aa86776b543979f9fcf4591cc03d4 2023-04-10 02:55:00,758 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/737aa86776b543979f9fcf4591cc03d4, entries=150, sequenceid=127, filesize=11.6 K 2023-04-10 02:55:00,759 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/9de45e5f092a46c1a9eed75f6e6596d8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9de45e5f092a46c1a9eed75f6e6596d8 2023-04-10 02:55:00,763 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9de45e5f092a46c1a9eed75f6e6596d8, entries=150, sequenceid=127, filesize=11.6 K 2023-04-10 02:55:00,764 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/5f5fccf3164945d7835edf4e24607247 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/5f5fccf3164945d7835edf4e24607247 2023-04-10 02:55:00,768 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/5f5fccf3164945d7835edf4e24607247, entries=150, sequenceid=127, filesize=11.6 K 2023-04-10 02:55:00,768 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for de5c1c1f03450ada9cd6207f5cfca947 in 920ms, sequenceid=127, compaction requested=false 2023-04-10 02:55:00,768 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:00,768 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:00,768 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:00,769 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:00,769 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,769 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:00,769 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:00,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,770 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:00,771 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:00,771 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:00,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:00,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@737ca7ce[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:00,773 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:00,773 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:00,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:00,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:00,774 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:00,774 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:00,774 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,774 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:00,774 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:00,776 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:00,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:00,778 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:00,778 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:00,778 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:00,779 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:00,779 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:00,785 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:00,785 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:00,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:00,785 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:00,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:00,785 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,787 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,790 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,790 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:00,790 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:00,790 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,790 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:00,790 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:00,790 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:00,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:00,790 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:00,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:00,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:00,790 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:00,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:00,790 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:00,790 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:00,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:00,890 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:00,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:00,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:00,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:00,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:00,894 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:00,894 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:00,894 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:00,894 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:00,894 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:00,894 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:00,895 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,896 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:00,896 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:00,896 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:00,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:00,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:00,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:00,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:00,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:00,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:00,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:00,897 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:00,897 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:00,897 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:00,898 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:00,898 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:00,898 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:00,898 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:00,898 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-10 02:55:00,898 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:00,898 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:00,899 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:00,899 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:00,899 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:00,899 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:01,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:01,081 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:01,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095361088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095361089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095361089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095361090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095361090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095361192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095361194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095361194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095361194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095361194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,308 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/62c20e6d39994490a3cc512fe27e1a24 2023-04-10 02:55:01,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095361394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095361397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095361397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095361397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095361398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095361698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095361699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095361700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095361700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:01,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095361701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,720 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/33bfa16911a74c33a4f3d7e88621f208 2023-04-10 02:55:01,778 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/668d42727eb84e1b8cbb9f45acaf63c4 2023-04-10 02:55:01,784 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/62c20e6d39994490a3cc512fe27e1a24 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/62c20e6d39994490a3cc512fe27e1a24 2023-04-10 02:55:01,789 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/62c20e6d39994490a3cc512fe27e1a24, entries=150, sequenceid=145, filesize=11.8 K 2023-04-10 02:55:01,790 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/33bfa16911a74c33a4f3d7e88621f208 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/33bfa16911a74c33a4f3d7e88621f208 2023-04-10 02:55:01,800 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/33bfa16911a74c33a4f3d7e88621f208, entries=150, sequenceid=145, filesize=11.8 K 2023-04-10 02:55:01,801 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/668d42727eb84e1b8cbb9f45acaf63c4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/668d42727eb84e1b8cbb9f45acaf63c4 2023-04-10 02:55:01,809 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/668d42727eb84e1b8cbb9f45acaf63c4, entries=150, sequenceid=145, filesize=11.8 K 2023-04-10 02:55:01,810 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for de5c1c1f03450ada9cd6207f5cfca947 in 911ms, sequenceid=145, compaction requested=true 2023-04-10 02:55:01,810 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:01,810 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:01,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:01,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:01,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:01,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:01,812 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:01,813 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:01,813 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:01,813 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:01,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:01,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0493eb[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:01,815 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:01,815 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:01,817 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:01,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:01,817 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:01,817 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:01,817 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:01,818 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,818 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:01,819 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:01,819 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:01,819 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:01,820 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:01,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:01,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:01,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:01,823 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:01,823 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:01,823 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:01,823 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:01,823 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:01,825 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:01,825 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:01,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:01,825 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:01,825 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:01,827 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:01,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:01,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:01,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:01,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:01,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:01,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:01,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:01,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:01,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:01,830 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:01,830 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:01,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:01,830 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:01,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:01,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:01,930 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:01,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:01,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:01,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:01,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:01,934 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:01,934 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:01,934 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:01,934 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:01,934 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:01,934 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:01,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,937 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:01,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:01,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:01,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:01,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:01,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:01,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:01,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:01,939 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:01,939 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:01,939 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:01,939 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:01,939 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:01,939 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:01,939 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:01,939 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-10 02:55:01,940 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:01,940 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:01,940 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:01,940 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:01,940 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:01,940 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:02,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:02,201 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:02,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095362212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095362212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095362213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095362214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095362214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095362318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095362318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095362319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095362319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095362320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,349 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/3b7775afd48e4943941fab02b7e82caa 2023-04-10 02:55:02,374 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/1f1427f1147b4db394e6488062fdcc2a 2023-04-10 02:55:02,398 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/50b1a57123b544a78adce8fd8f501471 2023-04-10 02:55:02,404 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/3b7775afd48e4943941fab02b7e82caa as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/3b7775afd48e4943941fab02b7e82caa 2023-04-10 02:55:02,409 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/3b7775afd48e4943941fab02b7e82caa, entries=150, sequenceid=162, filesize=11.8 K 2023-04-10 02:55:02,410 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/1f1427f1147b4db394e6488062fdcc2a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/1f1427f1147b4db394e6488062fdcc2a 2023-04-10 02:55:02,416 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/1f1427f1147b4db394e6488062fdcc2a, entries=150, sequenceid=162, filesize=11.8 K 2023-04-10 02:55:02,417 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/50b1a57123b544a78adce8fd8f501471 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/50b1a57123b544a78adce8fd8f501471 2023-04-10 02:55:02,422 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/50b1a57123b544a78adce8fd8f501471, entries=150, sequenceid=162, filesize=11.8 K 2023-04-10 02:55:02,423 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for de5c1c1f03450ada9cd6207f5cfca947 in 484ms, sequenceid=162, compaction requested=true 2023-04-10 02:55:02,423 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:02,423 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:02,423 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:02,423 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:02,424 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,424 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:02,424 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:02,425 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:02,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fa89442[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:02,427 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:02,427 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,429 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:02,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:02,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:02,429 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,429 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:02,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:02,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:02,433 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:02,433 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:02,433 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:02,433 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:02,433 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:02,436 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,436 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:02,436 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,437 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,439 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,441 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,441 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:02,442 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:02,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:02,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:02,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:02,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:02,442 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:02,442 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:02,441 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,442 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,442 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,442 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,442 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,442 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:02,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:02,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:02,523 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-10 02:55:02,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:02,524 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:02,524 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:02,524 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:02,524 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:02,524 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:02,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095362531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095362531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095362532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095362534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095362536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,542 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:02,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:02,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:02,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:02,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:02,546 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:02,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:02,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:02,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:02,547 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,547 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,552 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:02,552 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:02,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:02,553 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,553 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:02,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:02,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:02,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:02,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:02,554 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:02,554 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:02,554 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:02,555 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:02,555 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:02,555 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:02,555 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:02,560 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:02,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70a1da90[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:02,564 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:02,564 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,565 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,565 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:02,565 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:02,565 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:02,566 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,566 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:02,567 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,567 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:02,567 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,567 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:02,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:02,571 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:02,571 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:02,571 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:02,571 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:02,571 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:02,574 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,574 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:02,574 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,575 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,577 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,580 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:02,580 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:02,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:02,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,584 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:02,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:02,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:02,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:02,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:02,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,584 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:02,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,584 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:02,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095362637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095362637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095362637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095362638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095362640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,684 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:02,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:02,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:02,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:02,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:02,688 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:02,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:02,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:02,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:02,689 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,689 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,690 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:02,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:02,690 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:02,690 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,690 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:02,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:02,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:02,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:02,692 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:02,692 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:02,692 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:02,692 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:02,693 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,693 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:02,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f5519d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:02,695 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:02,695 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:02,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:02,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:02,697 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,697 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:02,699 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:02,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:02,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:02,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:02,701 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:02,701 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:02,701 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:02,701 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:02,701 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:02,703 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,703 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,703 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,703 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:02,707 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:02,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:02,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:02,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:02,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:02,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:02,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:02,708 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:02,708 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:02,808 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:02,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:02,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:02,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:02,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:02,811 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:02,811 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:02,811 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:02,811 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:02,812 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,812 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:02,814 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,815 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,815 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:02,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:02,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:02,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:02,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:02,816 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:02,816 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:02,816 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:02,816 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:02,816 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:02,816 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:02,816 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:02,816 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:02,816 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:02,817 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:02,817 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:02,817 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,817 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:02,817 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:02,818 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:02,820 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:02,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f071d63[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:02,820 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,821 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,821 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:02,821 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:02,821 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:02,821 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,821 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:02,822 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,822 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:02,822 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:02,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,822 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:02,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:02,825 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:02,826 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:02,826 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:02,826 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:02,826 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:02,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,828 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:02,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,829 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,830 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:02,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:02,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:02,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:02,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,832 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:02,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:02,833 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:02,833 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:02,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:02,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095362841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095362841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095362841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095362842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:02,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095362844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,933 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:02,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:02,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/540b3a73324844b0afd1d77c5ed3247c 2023-04-10 02:55:02,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:02,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:02,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:02,937 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:02,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:02,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:02,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:02,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,938 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,938 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:02,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:02,939 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,939 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:02,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:02,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:02,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:02,941 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:02,941 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:02,941 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:02,941 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:02,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:02,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:02,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:02,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:02,943 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,944 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,944 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:02,944 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:02,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44c68245[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:02,946 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:02,946 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:02,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:02,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:02,949 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,949 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:02,950 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:02,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:02,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:02,953 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:02,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:02,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:02,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:02,953 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:02,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:02,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,955 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,956 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,958 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:02,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:02,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:02,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:02,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:02,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:02,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:02,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:02,959 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:02,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:02,959 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:02,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:02,959 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:02,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:02,962 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/3b80eeac39c345678ac3be15d88f199d 2023-04-10 02:55:03,059 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:03,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:03,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:03,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:03,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:03,064 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:03,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:03,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:03,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,067 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,067 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,067 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:03,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:03,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:03,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:03,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:03,069 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,069 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:03,069 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:03,069 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:03,071 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:03,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a4da517[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:03,074 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:03,074 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,075 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,075 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,075 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:03,075 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:03,075 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:03,075 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:03,077 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:03,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:03,079 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:03,079 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:03,079 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:03,079 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:03,079 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:03,083 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,083 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,083 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,084 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,085 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,087 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,087 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,087 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,087 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,087 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:03,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:03,087 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,087 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,087 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:03,087 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,087 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:03,087 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,088 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095363145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095363146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095363146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095363147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095363149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,188 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:03,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:03,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:03,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:03,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:03,191 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:03,191 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,191 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:03,191 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:03,191 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,191 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,192 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,192 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,193 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,193 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:03,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:03,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:03,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:03,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:03,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:03,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:03,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:03,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:03,194 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,194 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:03,194 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:03,194 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:03,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:03,195 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,195 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:03,195 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:03,196 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,196 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,196 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,196 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:03,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:03,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7678727a[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:03,198 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:03,198 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,199 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:03,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:03,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:03,200 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,200 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:03,201 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,201 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,201 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,201 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:03,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:03,203 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:03,204 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:03,204 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:03,204 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:03,204 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:03,205 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,205 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,205 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,206 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,207 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,209 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,209 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,209 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,209 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,209 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,209 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:03,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:03,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:03,209 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,210 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:03,210 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:03,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,310 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:03,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:03,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:03,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:03,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:03,313 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:03,313 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,313 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:03,313 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:03,313 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,316 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,316 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,316 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,317 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:03,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:03,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:03,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:03,318 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:03,318 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,318 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:03,318 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:03,318 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:03,318 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,318 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:03,318 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:03,318 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,318 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:03,318 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:03,318 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,318 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:03,318 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:03,320 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,320 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,320 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,320 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:03,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@412385d3[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:03,322 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:03,322 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,323 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,323 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:03,323 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,323 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:03,323 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:03,323 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:03,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:03,328 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:03,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:03,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:03,331 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:03,331 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:03,331 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:03,331 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:03,331 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:03,332 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,332 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,332 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,333 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,334 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,336 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:03,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:03,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,336 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:03,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:03,336 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:03,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,372 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/cc231a18bfb44a1c8bb61c527688f271 2023-04-10 02:55:03,376 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/540b3a73324844b0afd1d77c5ed3247c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/540b3a73324844b0afd1d77c5ed3247c 2023-04-10 02:55:03,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/540b3a73324844b0afd1d77c5ed3247c, entries=200, sequenceid=179, filesize=14.1 K 2023-04-10 02:55:03,381 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/3b80eeac39c345678ac3be15d88f199d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/3b80eeac39c345678ac3be15d88f199d 2023-04-10 02:55:03,384 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/3b80eeac39c345678ac3be15d88f199d, entries=150, sequenceid=179, filesize=11.8 K 2023-04-10 02:55:03,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/cc231a18bfb44a1c8bb61c527688f271 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/cc231a18bfb44a1c8bb61c527688f271 2023-04-10 02:55:03,388 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/cc231a18bfb44a1c8bb61c527688f271, entries=150, sequenceid=179, filesize=11.8 K 2023-04-10 02:55:03,389 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for de5c1c1f03450ada9cd6207f5cfca947 in 866ms, sequenceid=179, compaction requested=true 2023-04-10 02:55:03,389 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:03,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:03,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:03,389 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:55:03,389 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:55:03,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:03,391 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62510 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:55:03,391 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60120 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:55:03,391 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/A is initiating minor compaction (all files) 2023-04-10 02:55:03,391 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/B is initiating minor compaction (all files) 2023-04-10 02:55:03,391 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/A in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,392 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/B in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,392 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/0a7bc07c08a047f886f1a8497f03a7ba, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/737aa86776b543979f9fcf4591cc03d4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/62c20e6d39994490a3cc512fe27e1a24, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/3b7775afd48e4943941fab02b7e82caa, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/540b3a73324844b0afd1d77c5ed3247c] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=61.0 K 2023-04-10 02:55:03,392 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/b0edc22730cf4ce99fa8ad45c93cb087, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9de45e5f092a46c1a9eed75f6e6596d8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/33bfa16911a74c33a4f3d7e88621f208, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/1f1427f1147b4db394e6488062fdcc2a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/3b80eeac39c345678ac3be15d88f199d] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=58.7 K 2023-04-10 02:55:03,392 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 0a7bc07c08a047f886f1a8497f03a7ba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681095298200 2023-04-10 02:55:03,392 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting b0edc22730cf4ce99fa8ad45c93cb087, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681095298200 2023-04-10 02:55:03,392 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 737aa86776b543979f9fcf4591cc03d4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1681095299333 2023-04-10 02:55:03,392 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 9de45e5f092a46c1a9eed75f6e6596d8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1681095299333 2023-04-10 02:55:03,393 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 62c20e6d39994490a3cc512fe27e1a24, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681095299955 2023-04-10 02:55:03,393 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 33bfa16911a74c33a4f3d7e88621f208, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681095299955 2023-04-10 02:55:03,393 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 3b7775afd48e4943941fab02b7e82caa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681095301088 2023-04-10 02:55:03,393 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 1f1427f1147b4db394e6488062fdcc2a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681095301088 2023-04-10 02:55:03,394 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 3b80eeac39c345678ac3be15d88f199d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681095302212 2023-04-10 02:55:03,394 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 540b3a73324844b0afd1d77c5ed3247c, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681095302209 2023-04-10 02:55:03,403 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#A#compaction#237 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:03,409 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#B#compaction#238 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:03,414 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/1bc7797a2c4d402ba5dc34037f63cab4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/1bc7797a2c4d402ba5dc34037f63cab4 2023-04-10 02:55:03,418 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/A of de5c1c1f03450ada9cd6207f5cfca947 into 1bc7797a2c4d402ba5dc34037f63cab4(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:03,418 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:03,418 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/A, priority=11, startTime=1681095303389; duration=0sec 2023-04-10 02:55:03,418 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:03,418 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:55:03,419 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60120 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:55:03,419 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/C is initiating minor compaction (all files) 2023-04-10 02:55:03,419 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/C in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,420 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/6515e6271a7f4b3eb3bdac25a58ea3a9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/5f5fccf3164945d7835edf4e24607247, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/668d42727eb84e1b8cbb9f45acaf63c4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/50b1a57123b544a78adce8fd8f501471, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/cc231a18bfb44a1c8bb61c527688f271] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=58.7 K 2023-04-10 02:55:03,420 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 6515e6271a7f4b3eb3bdac25a58ea3a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681095298200 2023-04-10 02:55:03,420 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 5f5fccf3164945d7835edf4e24607247, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1681095299333 2023-04-10 02:55:03,420 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 668d42727eb84e1b8cbb9f45acaf63c4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681095299955 2023-04-10 02:55:03,421 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 50b1a57123b544a78adce8fd8f501471, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681095301088 2023-04-10 02:55:03,421 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting cc231a18bfb44a1c8bb61c527688f271, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681095302212 2023-04-10 02:55:03,437 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:03,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:03,437 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#C#compaction#239 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:03,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:03,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:03,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:03,444 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:03,444 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,444 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:03,444 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:03,444 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,445 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,446 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,446 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,446 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,446 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:03,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:03,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:03,456 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:03,456 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,456 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:03,456 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:03,456 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:03,456 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,456 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:03,457 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-10 02:55:03,457 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:03,457 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:03,457 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:03,457 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:03,457 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:03,457 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:03,463 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/02739fce8b7d4cdd8b3525962fc4f9e9 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/02739fce8b7d4cdd8b3525962fc4f9e9 2023-04-10 02:55:03,468 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/C of de5c1c1f03450ada9cd6207f5cfca947 into 02739fce8b7d4cdd8b3525962fc4f9e9(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:03,468 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:03,468 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/C, priority=11, startTime=1681095303389; duration=0sec 2023-04-10 02:55:03,468 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:03,473 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/fa8ee734ecb144c695a3b1d570ffdbda 2023-04-10 02:55:03,482 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/f4f4921fc68f477492233eb530f27bfa 2023-04-10 02:55:03,491 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/be27b1c874414f1186da1c314d2bda04 2023-04-10 02:55:03,494 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/fa8ee734ecb144c695a3b1d570ffdbda as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/fa8ee734ecb144c695a3b1d570ffdbda 2023-04-10 02:55:03,498 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/fa8ee734ecb144c695a3b1d570ffdbda, entries=150, sequenceid=195, filesize=11.8 K 2023-04-10 02:55:03,498 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/f4f4921fc68f477492233eb530f27bfa as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/f4f4921fc68f477492233eb530f27bfa 2023-04-10 02:55:03,502 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/f4f4921fc68f477492233eb530f27bfa, entries=150, sequenceid=195, filesize=11.8 K 2023-04-10 02:55:03,502 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/be27b1c874414f1186da1c314d2bda04 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/be27b1c874414f1186da1c314d2bda04 2023-04-10 02:55:03,507 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/be27b1c874414f1186da1c314d2bda04, entries=150, sequenceid=195, filesize=11.8 K 2023-04-10 02:55:03,508 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for de5c1c1f03450ada9cd6207f5cfca947 in 52ms, sequenceid=195, compaction requested=false 2023-04-10 02:55:03,508 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:03,508 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,511 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:03,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3187f318[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:03,513 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:03,513 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,514 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,514 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:03,514 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:03,514 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:03,514 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,514 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:03,515 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,515 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:03,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:03,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:03,518 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:03,518 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:03,518 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:03,518 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:03,518 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:03,520 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,520 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,520 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,521 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,524 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,526 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,527 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,527 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,527 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,527 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,527 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,527 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:03,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:03,527 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,527 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,527 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,527 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:03,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,527 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:03,628 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:03,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:03,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:03,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:03,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:03,631 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:03,631 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,631 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:03,631 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:03,631 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,631 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,633 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,633 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,633 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,633 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:03,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:03,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:03,634 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:03,634 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,634 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:03,635 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:03,635 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:03,635 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,635 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:03,635 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:03,635 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,635 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:03,635 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:03,635 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,635 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:03,635 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:03,637 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,637 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,637 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,637 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:03,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66358d22[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:03,638 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:03,639 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,640 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:03,640 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:03,640 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:03,640 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,640 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:03,642 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:03,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:03,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:03,644 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:03,644 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:03,644 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:03,644 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:03,644 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:03,646 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,646 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,646 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,646 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,648 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,650 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,650 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,650 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:03,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:03,650 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,650 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,650 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,650 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:03,650 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:03,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:03,650 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,650 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,650 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,651 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:03,656 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:55:03,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:03,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:03,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:03,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:03,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:03,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:03,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095363667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095363668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095363670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095363670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095363670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,750 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:03,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:03,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:03,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:03,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:03,754 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:03,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:03,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:03,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,756 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,756 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,756 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:03,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:03,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:03,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:03,757 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,758 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:03,758 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:03,758 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,758 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:03,758 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:03,758 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,758 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:03,758 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,760 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:03,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:03,762 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:03,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53ebd83c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:03,762 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,763 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,763 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:03,763 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,763 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:03,763 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:03,763 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:03,764 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:03,764 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,764 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,764 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:03,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:03,767 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:03,767 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:03,767 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:03,767 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:03,767 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:03,768 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,768 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,769 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,770 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095363771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095363772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095363774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095363775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095363775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,778 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,778 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:03,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:03,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,778 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,778 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:03,778 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:03,778 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,778 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,778 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:03,778 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,817 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/a533c09634734945825bf4fc4076a7e3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a533c09634734945825bf4fc4076a7e3 2023-04-10 02:55:03,822 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/B of de5c1c1f03450ada9cd6207f5cfca947 into a533c09634734945825bf4fc4076a7e3(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:03,822 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:03,822 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/B, priority=11, startTime=1681095303389; duration=0sec 2023-04-10 02:55:03,822 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:03,878 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:03,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:03,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:03,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:03,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:03,888 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:03,888 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,888 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:03,888 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:03,888 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,889 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,890 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,890 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,890 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,890 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:03,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:03,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:03,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:03,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:03,893 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:03,893 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:03,893 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:03,894 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:03,894 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:03,894 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,894 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:03,894 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:03,894 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:03,894 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:03,894 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:03,894 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,894 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:03,894 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:03,896 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,898 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:03,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ebbd28f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:03,898 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:03,898 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,898 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,898 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:03,899 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,900 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:03,902 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,903 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:03,903 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:03,903 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:03,905 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,905 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:03,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,905 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,905 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:03,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:03,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:03,910 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:03,910 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:03,910 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:03,910 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:03,910 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:03,911 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,911 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,911 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:03,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:03,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,913 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,914 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:03,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:03,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:03,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:03,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:03,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:03,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:03,916 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:03,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:03,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:03,917 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:03,917 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:03,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:03,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095363974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095363975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095363978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:03,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095363979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:03,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095363979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,017 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:04,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:04,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:04,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:04,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:04,020 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:04,020 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,020 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:04,020 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:04,021 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,021 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,022 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:04,022 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,023 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,023 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:04,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:04,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:04,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:04,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:04,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:04,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:04,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:04,024 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,024 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:04,024 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:04,024 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:04,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:04,025 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,025 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:04,025 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:04,027 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,028 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,028 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:04,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:04,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aaef73[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:04,029 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:04,029 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,030 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:04,030 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:04,030 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:04,030 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,031 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:04,032 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,032 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:04,032 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:04,032 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:04,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:04,034 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:04,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:04,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:04,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:04,034 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:04,035 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,035 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,036 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:04,036 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,040 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,040 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,040 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:04,040 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:04,040 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:04,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:04,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:04,040 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:04,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:04,040 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,040 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,040 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:04,040 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,041 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:04,041 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:04,084 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/845efcfc8edc45c3a8194f0e9752fcf1 2023-04-10 02:55:04,141 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:04,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:04,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:04,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:04,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:04,144 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:04,144 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,144 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:04,144 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:04,145 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,145 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,146 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:04,146 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:04,146 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,147 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:04,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:04,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:04,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:04,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:04,148 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:04,148 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,148 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:04,148 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:04,148 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:04,148 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,148 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:04,148 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:04,148 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,148 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:04,148 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:04,148 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,148 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:04,148 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:04,150 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,150 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,150 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:04,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4abc3ed9[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:04,151 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:04,151 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,152 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,153 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:04,153 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:04,153 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:04,153 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,153 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:04,155 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:04,155 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,155 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:04,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,155 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:04,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:04,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:04,157 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:04,157 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:04,157 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:04,157 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:04,157 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:04,159 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,159 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,159 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,160 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,161 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:04,165 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:04,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:04,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:04,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:04,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:04,166 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,166 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,166 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,166 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,166 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,166 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,166 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:04,166 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:04,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:04,266 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:04,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:04,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:04,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:04,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:04,269 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:04,269 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,269 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:04,269 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:04,270 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,270 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,272 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:04,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:04,272 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,272 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:04,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:04,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:04,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:04,273 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:04,273 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,273 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:04,273 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:04,274 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:04,274 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,274 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:04,274 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:04,274 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,274 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:04,274 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:04,274 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,274 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:04,274 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:04,275 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,275 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,276 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,276 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:04,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@471b358a[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:04,277 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:04,277 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095364277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,279 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,279 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:04,279 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:04,279 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:04,279 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,279 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:04,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095364278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,280 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,280 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:04,280 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,280 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:04,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:04,283 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:04,283 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:04,283 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:04,283 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:04,283 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:04,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095364283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095364283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095364283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,286 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,286 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:04,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,286 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:04,286 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,287 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:04,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:04,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,290 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:04,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:04,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:04,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:04,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:04,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:04,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:04,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:04,291 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:04,291 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:04,391 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:04,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:04,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:04,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:04,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:04,395 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:04,396 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,396 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:04,396 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:04,396 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,396 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,398 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:04,398 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:04,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:04,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:04,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:04,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:04,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:04,400 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:04,400 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,400 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:04,400 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:04,400 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:04,400 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,400 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:04,400 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:04,400 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,400 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:04,401 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:04,401 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,401 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:04,401 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:04,402 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,402 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,402 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,403 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:04,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:04,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1764b84c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:04,404 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:04,404 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,406 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:04,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:04,406 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,407 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:04,407 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:04,408 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,408 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:04,409 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:04,409 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:04,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:04,411 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:04,412 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:04,412 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:04,412 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:04,412 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:04,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:04,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:04,414 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,415 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,417 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:04,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:04,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:04,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:04,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:04,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:04,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:04,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,418 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:04,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:04,418 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:04,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:04,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:04,495 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/8b78ac654ad446e5aec132835bb3442b 2023-04-10 02:55:04,518 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:04,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:04,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:04,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:04,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:04,535 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:04,536 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,536 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:04,536 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:04,536 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/95adb0e52d184ab8b05d417193a05992 2023-04-10 02:55:04,537 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,537 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,539 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:04,539 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:04,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:04,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,539 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,539 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:04,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:04,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/845efcfc8edc45c3a8194f0e9752fcf1 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/845efcfc8edc45c3a8194f0e9752fcf1 2023-04-10 02:55:04,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:04,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:04,552 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:04,552 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,552 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:04,552 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:04,552 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:04,553 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,553 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:04,553 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:04,554 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:04,555 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,557 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/845efcfc8edc45c3a8194f0e9752fcf1, entries=200, sequenceid=209, filesize=14.1 K 2023-04-10 02:55:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,557 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/8b78ac654ad446e5aec132835bb3442b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/8b78ac654ad446e5aec132835bb3442b 2023-04-10 02:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:04,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40621ca4[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:04,559 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:04,559 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,560 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:04,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:04,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:04,561 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,561 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:04,562 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,562 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:04,562 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:04,562 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:04,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/8b78ac654ad446e5aec132835bb3442b, entries=150, sequenceid=209, filesize=11.8 K 2023-04-10 02:55:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,563 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/95adb0e52d184ab8b05d417193a05992 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/95adb0e52d184ab8b05d417193a05992 2023-04-10 02:55:04,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:04,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:04,565 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:04,565 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:04,565 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:04,565 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:04,565 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:04,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:04,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,570 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,571 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/95adb0e52d184ab8b05d417193a05992, entries=150, sequenceid=209, filesize=11.8 K 2023-04-10 02:55:04,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,572 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for de5c1c1f03450ada9cd6207f5cfca947 in 916ms, sequenceid=209, compaction requested=true 2023-04-10 02:55:04,572 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,573 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:04,573 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,573 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:04,573 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:04,573 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:04,574 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38908 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:04,574 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/A is initiating minor compaction (all files) 2023-04-10 02:55:04,574 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/A in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,574 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/1bc7797a2c4d402ba5dc34037f63cab4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/fa8ee734ecb144c695a3b1d570ffdbda, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/845efcfc8edc45c3a8194f0e9752fcf1] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=38.0 K 2023-04-10 02:55:04,574 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 1bc7797a2c4d402ba5dc34037f63cab4, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681095302212 2023-04-10 02:55:04,574 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:04,575 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/B is initiating minor compaction (all files) 2023-04-10 02:55:04,575 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/B in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,575 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,575 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a533c09634734945825bf4fc4076a7e3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/f4f4921fc68f477492233eb530f27bfa, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/8b78ac654ad446e5aec132835bb3442b] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=35.7 K 2023-04-10 02:55:04,575 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting fa8ee734ecb144c695a3b1d570ffdbda, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1681095302528 2023-04-10 02:55:04,575 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:04,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:04,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:04,575 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,575 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:04,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:04,575 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:04,575 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:04,575 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,575 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting a533c09634734945825bf4fc4076a7e3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681095302212 2023-04-10 02:55:04,575 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 845efcfc8edc45c3a8194f0e9752fcf1, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681095303653 2023-04-10 02:55:04,575 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:04,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:04,575 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,576 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,576 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,576 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting f4f4921fc68f477492233eb530f27bfa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1681095302528 2023-04-10 02:55:04,576 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,576 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 8b78ac654ad446e5aec132835bb3442b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681095303653 2023-04-10 02:55:04,585 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#B#compaction#246 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:04,589 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#A#compaction#247 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:04,660 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/721734ef1ce348b69b84ffecb11e88f2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/721734ef1ce348b69b84ffecb11e88f2 2023-04-10 02:55:04,664 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/A of de5c1c1f03450ada9cd6207f5cfca947 into 721734ef1ce348b69b84ffecb11e88f2(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:04,664 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:04,664 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/A, priority=13, startTime=1681095304573; duration=0sec 2023-04-10 02:55:04,665 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:04,665 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:04,666 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:04,666 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/C is initiating minor compaction (all files) 2023-04-10 02:55:04,666 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/C in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,666 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/02739fce8b7d4cdd8b3525962fc4f9e9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/be27b1c874414f1186da1c314d2bda04, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/95adb0e52d184ab8b05d417193a05992] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=35.7 K 2023-04-10 02:55:04,667 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 02739fce8b7d4cdd8b3525962fc4f9e9, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681095302212 2023-04-10 02:55:04,667 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting be27b1c874414f1186da1c314d2bda04, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1681095302528 2023-04-10 02:55:04,667 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 95adb0e52d184ab8b05d417193a05992, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681095303653 2023-04-10 02:55:04,674 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#C#compaction#248 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:04,676 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:04,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:04,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:04,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:04,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:04,684 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:04,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:04,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:04,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,693 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:04,693 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:04,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:04,693 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,694 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:04,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:04,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:04,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:04,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:04,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:04,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:04,695 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,695 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:04,696 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-10 02:55:04,696 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:04,696 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:04,696 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:04,696 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:04,696 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:04,696 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:04,707 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/fd347fcead4947209674a49ba3b0f679 2023-04-10 02:55:04,726 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/6ae99ed40ada4eb8aba6852d33312217 2023-04-10 02:55:04,736 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/c26733909493467e95f6c235b696dcd5 2023-04-10 02:55:04,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/fd347fcead4947209674a49ba3b0f679 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/fd347fcead4947209674a49ba3b0f679 2023-04-10 02:55:04,745 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/fd347fcead4947209674a49ba3b0f679, entries=150, sequenceid=231, filesize=11.8 K 2023-04-10 02:55:04,745 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/6ae99ed40ada4eb8aba6852d33312217 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/6ae99ed40ada4eb8aba6852d33312217 2023-04-10 02:55:04,750 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/6ae99ed40ada4eb8aba6852d33312217, entries=150, sequenceid=231, filesize=11.8 K 2023-04-10 02:55:04,751 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/c26733909493467e95f6c235b696dcd5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/c26733909493467e95f6c235b696dcd5 2023-04-10 02:55:04,756 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/c26733909493467e95f6c235b696dcd5, entries=150, sequenceid=231, filesize=11.8 K 2023-04-10 02:55:04,757 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for de5c1c1f03450ada9cd6207f5cfca947 in 61ms, sequenceid=231, compaction requested=false 2023-04-10 02:55:04,757 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:04,757 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:04,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:04,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:04,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:04,759 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,759 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:04,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:04,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b03b887[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:04,762 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:04,762 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,763 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,763 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:04,763 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:04,763 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:04,763 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,763 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:04,764 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:04,764 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:04,764 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,764 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:04,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:04,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:04,768 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:04,768 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:04,768 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:04,768 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:04,768 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:04,769 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:04,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:04,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:04,771 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,772 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,778 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:04,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:04,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:04,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:04,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:04,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:04,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:04,779 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:04,779 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:04,779 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:04,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:04,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:04,792 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:55:04,793 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:04,793 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:04,793 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:04,793 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:04,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:04,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:04,805 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/38d5595578f241009da2d4747b4bd0a8 2023-04-10 02:55:04,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095364807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095364808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095364812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095364812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095364812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,818 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/9791942f97e249cdb2c726dc1ab4682e 2023-04-10 02:55:04,828 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/0614b876287244478bd88c55b895cb36 2023-04-10 02:55:04,832 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/38d5595578f241009da2d4747b4bd0a8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/38d5595578f241009da2d4747b4bd0a8 2023-04-10 02:55:04,835 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/38d5595578f241009da2d4747b4bd0a8, entries=200, sequenceid=243, filesize=14.1 K 2023-04-10 02:55:04,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/9791942f97e249cdb2c726dc1ab4682e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9791942f97e249cdb2c726dc1ab4682e 2023-04-10 02:55:04,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9791942f97e249cdb2c726dc1ab4682e, entries=150, sequenceid=243, filesize=11.8 K 2023-04-10 02:55:04,840 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/0614b876287244478bd88c55b895cb36 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0614b876287244478bd88c55b895cb36 2023-04-10 02:55:04,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0614b876287244478bd88c55b895cb36, entries=150, sequenceid=243, filesize=11.8 K 2023-04-10 02:55:04,851 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for de5c1c1f03450ada9cd6207f5cfca947 in 58ms, sequenceid=243, compaction requested=true 2023-04-10 02:55:04,851 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:04,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:04,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-10 02:55:04,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-10 02:55:04,879 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:04,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:04,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:04,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:04,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:04,883 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:04,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:04,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:04,884 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,884 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,885 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:04,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:04,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:04,885 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,886 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,886 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:04,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:04,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:04,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:04,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:04,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:04,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:04,892 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:04,892 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:04,892 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:04,898 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:04,898 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:04,899 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:04,899 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:04,899 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-10 02:55:04,899 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:04,899 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:04,899 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:04,899 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:04,899 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:04,899 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:04,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:04,916 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:04,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4442) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4381) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4857) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4851) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4847) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3159) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095364923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095364924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095364925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095364928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:04,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:04,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095364928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095365030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095365030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095365032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095365032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095365034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,059 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/54c12902a1864049a362f85ea3bd722a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/54c12902a1864049a362f85ea3bd722a 2023-04-10 02:55:05,085 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/B of de5c1c1f03450ada9cd6207f5cfca947 into 54c12902a1864049a362f85ea3bd722a(size=12.3 K), total size for store is 35.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:05,085 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:05,085 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/B, priority=13, startTime=1681095304573; duration=0sec 2023-04-10 02:55:05,085 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-10 02:55:05,085 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-10 02:55:05,088 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-10 02:55:05,088 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-10 02:55:05,088 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. because compaction request was cancelled 2023-04-10 02:55:05,088 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:05,091 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36620 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:05,091 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/B is initiating minor compaction (all files) 2023-04-10 02:55:05,091 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/B in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:05,091 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/54c12902a1864049a362f85ea3bd722a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/6ae99ed40ada4eb8aba6852d33312217, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9791942f97e249cdb2c726dc1ab4682e] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=35.8 K 2023-04-10 02:55:05,091 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 54c12902a1864049a362f85ea3bd722a, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681095303653 2023-04-10 02:55:05,092 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6ae99ed40ada4eb8aba6852d33312217, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681095303666 2023-04-10 02:55:05,093 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 9791942f97e249cdb2c726dc1ab4682e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681095304788 2023-04-10 02:55:05,100 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/eae585f77f8a448c8c0c54efbac11d99 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/eae585f77f8a448c8c0c54efbac11d99 2023-04-10 02:55:05,106 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/C of de5c1c1f03450ada9cd6207f5cfca947 into eae585f77f8a448c8c0c54efbac11d99(size=12.3 K), total size for store is 35.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:05,106 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:05,106 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/C, priority=13, startTime=1681095304573; duration=0sec 2023-04-10 02:55:05,106 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:05,106 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:05,108 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#B#compaction#256 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:05,110 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39010 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:05,110 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/A is initiating minor compaction (all files) 2023-04-10 02:55:05,110 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/A in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:05,110 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/721734ef1ce348b69b84ffecb11e88f2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/fd347fcead4947209674a49ba3b0f679, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/38d5595578f241009da2d4747b4bd0a8] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=38.1 K 2023-04-10 02:55:05,112 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 721734ef1ce348b69b84ffecb11e88f2, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681095303653 2023-04-10 02:55:05,113 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting fd347fcead4947209674a49ba3b0f679, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681095303666 2023-04-10 02:55:05,114 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 38d5595578f241009da2d4747b4bd0a8, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681095304788 2023-04-10 02:55:05,133 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#A#compaction#257 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:05,152 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/11445e9958a2444ead4f83441c91f3a8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/11445e9958a2444ead4f83441c91f3a8 2023-04-10 02:55:05,159 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/A of de5c1c1f03450ada9cd6207f5cfca947 into 11445e9958a2444ead4f83441c91f3a8(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:05,159 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:05,159 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/A, priority=13, startTime=1681095304851; duration=0sec 2023-04-10 02:55:05,159 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:05,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095365234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095365235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095365235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095365249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095365249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,326 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/8745bb7edb7040d4b24def654ccc05ea 2023-04-10 02:55:05,535 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/5389ca2900d24009b78f42e000b428f3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/5389ca2900d24009b78f42e000b428f3 2023-04-10 02:55:05,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40360 deadline: 1681095365537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40414 deadline: 1681095365537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,540 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/B of de5c1c1f03450ada9cd6207f5cfca947 into 5389ca2900d24009b78f42e000b428f3(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:05,540 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:05,540 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/B, priority=13, startTime=1681095304851; duration=0sec 2023-04-10 02:55:05,540 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:05,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40424 deadline: 1681095365540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1681095365553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:05,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095365553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,738 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/25ebf710be6246b495956ba2b2c3303d 2023-04-10 02:55:05,751 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/b43bff405a354a5eb6b5e75971f7e08d 2023-04-10 02:55:05,755 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/8745bb7edb7040d4b24def654ccc05ea as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/8745bb7edb7040d4b24def654ccc05ea 2023-04-10 02:55:05,758 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/8745bb7edb7040d4b24def654ccc05ea, entries=150, sequenceid=263, filesize=11.9 K 2023-04-10 02:55:05,759 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/25ebf710be6246b495956ba2b2c3303d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/25ebf710be6246b495956ba2b2c3303d 2023-04-10 02:55:05,770 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/25ebf710be6246b495956ba2b2c3303d, entries=150, sequenceid=263, filesize=11.9 K 2023-04-10 02:55:05,773 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/b43bff405a354a5eb6b5e75971f7e08d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/b43bff405a354a5eb6b5e75971f7e08d 2023-04-10 02:55:05,785 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/b43bff405a354a5eb6b5e75971f7e08d, entries=150, sequenceid=263, filesize=11.9 K 2023-04-10 02:55:05,786 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for de5c1c1f03450ada9cd6207f5cfca947 in 887ms, sequenceid=263, compaction requested=true 2023-04-10 02:55:05,786 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:05,786 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:05,788 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:05,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:05,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:05,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:05,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@752a7708[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:05,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:05,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:05,791 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:05,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:05,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:05,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:05,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:05,792 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,792 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:05,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:05,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:05,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:05,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:05,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:05,797 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:05,798 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:05,798 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:05,798 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:05,798 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:05,800 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:05,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:05,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:05,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:05,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:05,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:05,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:05,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:05,802 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:05,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:05,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:05,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:05,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,807 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:05,807 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:05,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:05,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:05,808 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,808 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:05,808 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:05,808 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:05,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:05,808 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:05,808 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:05,808 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:05,808 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:05,808 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:05,808 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:05,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:05,908 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:05,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:05,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:05,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:05,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:05,911 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:05,912 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:05,912 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:05,912 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:05,912 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:05,912 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:05,913 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:05,913 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:05,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:05,914 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:05,914 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:05,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:05,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:05,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:05,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:05,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:05,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:05,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:05,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:05,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:05,915 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:05,915 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:05,916 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:55:05,916 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:05,916 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:05,916 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:05,916 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:05,916 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:05,916 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:06,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:06,043 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:06,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095306056, max timestamp=1681095306061Num uniques 859; 2023-04-10 02:55:06,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:55:06,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095306056, max timestamp=1681095306061Num uniques 859; 2023-04-10 02:55:06,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095306056, max timestamp=1681095306061Num uniques 859; 2023-04-10 02:55:06,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:55:06,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:55:06,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40390 deadline: 1681095366072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095366072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40414 deadline: 1681095366073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1681095366073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095366074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095366177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40390 deadline: 1681095366177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40414 deadline: 1681095366177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1681095366177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095366178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,323 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/4f3579f2c8e64be7988d194d46c1dae5 2023-04-10 02:55:06,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40390 deadline: 1681095366379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095366379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40414 deadline: 1681095366381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1681095366381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095366381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40390 deadline: 1681095366683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095366683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1681095366684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40414 deadline: 1681095366686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:06,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095366686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,734 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/d2758ac3ffed43fc847d74076898a662 2023-04-10 02:55:06,760 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/3ef771c4bb984206b377e12d3b7b9dd6 2023-04-10 02:55:06,766 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/4f3579f2c8e64be7988d194d46c1dae5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/4f3579f2c8e64be7988d194d46c1dae5 2023-04-10 02:55:06,775 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/4f3579f2c8e64be7988d194d46c1dae5, entries=150, sequenceid=279, filesize=11.9 K 2023-04-10 02:55:06,776 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/d2758ac3ffed43fc847d74076898a662 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/d2758ac3ffed43fc847d74076898a662 2023-04-10 02:55:06,780 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/d2758ac3ffed43fc847d74076898a662, entries=150, sequenceid=279, filesize=11.9 K 2023-04-10 02:55:06,781 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/3ef771c4bb984206b377e12d3b7b9dd6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/3ef771c4bb984206b377e12d3b7b9dd6 2023-04-10 02:55:06,785 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/3ef771c4bb984206b377e12d3b7b9dd6, entries=150, sequenceid=279, filesize=11.9 K 2023-04-10 02:55:06,786 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for de5c1c1f03450ada9cd6207f5cfca947 in 870ms, sequenceid=279, compaction requested=true 2023-04-10 02:55:06,786 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:06,786 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:06,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:06,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:06,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:06,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:06,787 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:06,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:06,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:06,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:06,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47be0739[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:06,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:06,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:06,791 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:06,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:06,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:06,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:06,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:06,791 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,791 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:06,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:06,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:06,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:06,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:06,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:06,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:06,797 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:06,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:06,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:06,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:06,797 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:06,799 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:06,799 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:06,799 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:06,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:06,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:06,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:06,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:06,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:06,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,802 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:06,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:06,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:06,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:06,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:06,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:06,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:06,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:06,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:06,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:06,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:06,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:06,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:06,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:06,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:06,805 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:06,804 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:06,805 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:06,905 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:06,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:06,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:06,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:06,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:06,913 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:06,913 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:06,913 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:06,913 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:06,913 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:06,913 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:06,915 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:06,915 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:06,915 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,915 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:06,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:06,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:06,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:06,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:06,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:06,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:06,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:06,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:06,918 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:06,918 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:06,918 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:06,919 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=120.76 KB heapSize=335.77 KB 2023-04-10 02:55:06,919 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:06,919 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:06,919 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:06,919 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:06,919 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:06,919 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:06,930 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/5e2c953b053148d4a80fab6b65e8126d 2023-04-10 02:55:06,949 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/997c814577ee4482b6bc756071a36196 2023-04-10 02:55:06,962 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/26c47941beb34ee9b728dc68e0b3d9ff 2023-04-10 02:55:06,968 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/5e2c953b053148d4a80fab6b65e8126d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/5e2c953b053148d4a80fab6b65e8126d 2023-04-10 02:55:06,974 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/5e2c953b053148d4a80fab6b65e8126d, entries=150, sequenceid=312, filesize=11.9 K 2023-04-10 02:55:06,980 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/997c814577ee4482b6bc756071a36196 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/997c814577ee4482b6bc756071a36196 2023-04-10 02:55:06,983 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/997c814577ee4482b6bc756071a36196, entries=150, sequenceid=312, filesize=11.9 K 2023-04-10 02:55:06,984 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/26c47941beb34ee9b728dc68e0b3d9ff as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/26c47941beb34ee9b728dc68e0b3d9ff 2023-04-10 02:55:06,988 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/26c47941beb34ee9b728dc68e0b3d9ff, entries=150, sequenceid=312, filesize=11.9 K 2023-04-10 02:55:06,989 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~335.72 KB/343776, currentSize=0 B/0 for de5c1c1f03450ada9cd6207f5cfca947 in 71ms, sequenceid=312, compaction requested=true 2023-04-10 02:55:06,989 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:06,989 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:06,989 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:06,990 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:06,990 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,990 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:06,990 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:06,995 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:06,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:06,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:06,997 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:06,997 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:06,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:06,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:06,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:06,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:06,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:06,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:06,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17e9b456[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:06,999 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:06,999 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,001 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,001 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:07,001 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:07,001 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:07,001 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,001 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:07,003 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:07,003 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,003 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:07,003 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:07,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:07,007 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:07,007 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:07,007 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:07,007 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:07,007 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:07,009 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,009 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,009 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,010 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,011 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,015 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,015 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,015 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,015 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:07,015 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,015 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:07,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,015 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:07,015 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,015 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,015 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,016 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:07,016 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:07,116 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:07,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:07,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:07,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:07,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:07,119 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:07,119 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,119 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:07,119 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:07,119 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,119 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,121 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,121 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,121 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,121 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:07,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:07,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:07,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:07,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:07,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:07,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:07,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:07,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:07,122 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,122 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:07,122 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:07,122 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:07,123 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:07,123 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,123 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:07,123 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:07,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,124 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:07,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d8e1ac[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:07,126 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:07,126 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,128 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:07,128 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,128 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:07,130 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,130 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:07,130 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,130 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:07,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:07,132 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:07,132 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:07,132 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:07,132 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:07,132 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:07,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,135 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,136 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,138 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,138 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,138 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,138 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:07,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:07,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:07,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,138 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,139 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,138 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,139 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:07,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:07,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:55:07,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:07,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:07,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:07,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:07,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:07,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:07,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095367234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095367235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095307231, max timestamp=1681095307236Num uniques 859; 2023-04-10 02:55:07,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:07,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095307231, max timestamp=1681095307236Num uniques 859; 2023-04-10 02:55:07,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:07,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095307231, max timestamp=1681095307236Num uniques 859; 2023-04-10 02:55:07,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:07,239 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:07,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:07,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:07,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:07,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:07,242 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:07,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:07,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:07,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,243 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,244 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,244 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,244 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,244 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:07,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:07,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:07,245 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:07,245 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,245 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:07,245 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:07,245 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:07,245 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,245 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:07,246 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:07,246 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,246 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:07,246 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:07,246 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,246 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:07,246 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:07,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,248 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:07,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:07,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@476c48a8[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:07,250 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:07,250 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,251 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,251 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:07,251 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:07,251 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:07,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:07,252 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:07,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:07,255 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:07,255 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:07,255 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:07,255 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:07,255 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:07,256 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,256 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,256 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,257 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,258 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,262 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,262 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,262 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,262 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,262 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:07,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:07,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,262 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,262 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,262 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:07,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:07,262 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,262 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:07,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,262 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,262 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1681095367260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40390 deadline: 1681095367260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40414 deadline: 1681095367264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095367339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095367339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,362 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:07,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:07,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:07,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:07,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:07,366 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:07,366 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,366 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:07,366 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:07,367 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,367 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,368 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,368 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,368 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,368 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:07,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:07,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:07,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:07,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,369 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:07,369 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40390 deadline: 1681095367367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,369 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:07,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1681095367367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:07,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:07,370 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,370 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:07,370 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:07,370 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:07,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:07,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:07,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:07,372 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,372 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,372 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,372 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:07,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,373 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:07,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40414 deadline: 1681095367370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,373 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@705c08b3[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:07,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,375 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:07,375 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:07,375 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:07,375 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,375 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:07,384 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:07,384 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,384 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,384 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:07,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:07,387 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:07,387 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:07,387 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:07,387 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:07,387 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:07,388 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,388 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,388 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,389 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,390 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,392 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:07,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:07,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,393 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,393 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:07,393 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:07,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:07,493 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:07,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:07,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:07,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:07,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:07,497 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:07,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:07,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:07,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,498 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,499 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,499 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:07,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:07,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:07,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:07,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:07,500 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,500 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:07,500 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:07,500 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:07,501 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:07,501 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,501 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:07,501 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:07,503 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,503 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,503 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,503 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:07,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:07,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2acd59ea[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:07,505 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:07,505 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,506 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,506 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:07,506 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:07,506 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:07,506 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,506 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:07,507 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,507 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:07,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:07,510 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:07,510 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:07,510 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:07,510 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:07,510 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:07,512 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,512 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,512 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,512 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,514 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,517 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,517 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,517 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,517 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,517 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:07,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:07,517 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:07,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,518 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,518 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:07,518 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:07,518 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,518 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,518 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095367541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095367542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40390 deadline: 1681095367570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1681095367571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40414 deadline: 1681095367575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,612 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/f4c557dbec0f4d1389bb88e6f2629008 2023-04-10 02:55:07,618 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:07,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:07,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:07,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:07,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:07,622 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:07,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:07,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:07,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,623 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,623 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,624 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,624 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:07,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:07,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:07,625 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:07,625 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,625 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:07,625 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:07,625 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:07,625 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,625 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:07,625 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:07,625 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,625 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:07,625 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:07,625 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,625 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:07,625 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:07,627 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:07,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cbe089e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:07,628 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:07,629 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,630 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:07,630 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:07,630 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:07,630 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,630 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:07,632 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:07,632 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,632 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,632 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:07,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:07,635 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:07,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:07,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:07,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:07,635 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:07,636 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,636 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,636 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,637 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,638 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:07,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:07,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,640 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:07,640 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:07,641 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:07,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,741 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:07,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:07,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:07,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:07,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:07,745 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:07,745 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,745 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:07,745 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:07,746 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,746 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,747 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,747 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,747 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,747 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:07,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:07,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:07,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:07,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:07,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:07,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:07,748 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:07,748 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,748 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:07,749 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:07,749 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,749 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:07,749 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:07,749 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,749 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:07,749 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:07,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,751 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,751 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,751 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:07,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:07,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29b238a9[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:07,753 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:07,753 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,754 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:07,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:07,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:07,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:07,756 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,756 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:07,756 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,756 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:07,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:07,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:07,758 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:07,758 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:07,758 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:07,758 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:07,758 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:07,766 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,766 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,766 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,767 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,768 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,770 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:07,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:07,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,771 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,771 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,771 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:07,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:07,771 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:07,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095367846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095367846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,871 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:07,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:07,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:07,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:07,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:07,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,878 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:07,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40390 deadline: 1681095367876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,878 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,879 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:07,879 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:07,879 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,879 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,880 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1681095367879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:07,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:07,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:07,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40414 deadline: 1681095367879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,884 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,884 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,884 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:07,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:07,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:07,888 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:07,888 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:07,888 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:07,889 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:07,889 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:07,889 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,889 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:07,889 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:07,889 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:07,889 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:07,889 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:07,889 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:07,891 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,891 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,892 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,892 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:07,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39bba4bc[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:07,912 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:07,912 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,914 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:07,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:07,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:07,915 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,915 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:07,916 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:07,916 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,916 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:07,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:07,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:07,919 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:07,919 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:07,920 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:07,920 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:07,920 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:07,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:07,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,922 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:07,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:07,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:07,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:07,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:07,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:07,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:07,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:07,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:07,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:07,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:07,926 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:07,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,926 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:07,927 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:07,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:07,927 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:07,927 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:08,023 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/a276c6007d8044d8bff5f90c75b8acb4 2023-04-10 02:55:08,027 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:08,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:08,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:08,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:08,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:08,039 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:08,039 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:08,039 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:08,039 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:08,040 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,040 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,042 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:08,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:08,042 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:08,043 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,043 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:08,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:08,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:08,060 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:08,060 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:08,060 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:08,062 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:08,062 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:08,063 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:08,063 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:08,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/476c0e23750a4aeea31910c6744224c2 2023-04-10 02:55:08,070 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:08,070 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:08,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:08,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79f9b868[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:08,075 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:08,075 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,077 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:08,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/f4c557dbec0f4d1389bb88e6f2629008 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/f4c557dbec0f4d1389bb88e6f2629008 2023-04-10 02:55:08,079 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:08,079 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:08,079 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:08,079 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,080 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:08,080 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:08,080 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:08,080 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:08,080 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:08,080 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:08,080 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:08,083 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,083 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:08,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,083 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:08,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:08,084 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,086 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,089 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:08,089 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:08,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:08,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:08,090 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,090 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,090 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,090 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,090 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,090 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:08,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:08,090 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,090 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:08,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:08,090 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:08,090 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:08,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/f4c557dbec0f4d1389bb88e6f2629008, entries=150, sequenceid=324, filesize=11.9 K 2023-04-10 02:55:08,092 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/a276c6007d8044d8bff5f90c75b8acb4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a276c6007d8044d8bff5f90c75b8acb4 2023-04-10 02:55:08,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a276c6007d8044d8bff5f90c75b8acb4, entries=150, sequenceid=324, filesize=11.9 K 2023-04-10 02:55:08,100 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/476c0e23750a4aeea31910c6744224c2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/476c0e23750a4aeea31910c6744224c2 2023-04-10 02:55:08,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/476c0e23750a4aeea31910c6744224c2, entries=150, sequenceid=324, filesize=11.9 K 2023-04-10 02:55:08,106 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for de5c1c1f03450ada9cd6207f5cfca947 in 910ms, sequenceid=324, compaction requested=true 2023-04-10 02:55:08,106 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:08,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:08,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:08,106 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:55:08,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-10 02:55:08,107 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-10 02:55:08,111 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61394 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:55:08,111 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/A is initiating minor compaction (all files) 2023-04-10 02:55:08,111 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/A in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:08,111 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/11445e9958a2444ead4f83441c91f3a8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/8745bb7edb7040d4b24def654ccc05ea, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/4f3579f2c8e64be7988d194d46c1dae5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/5e2c953b053148d4a80fab6b65e8126d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/f4c557dbec0f4d1389bb88e6f2629008] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=60.0 K 2023-04-10 02:55:08,112 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 11445e9958a2444ead4f83441c91f3a8, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681095304788 2023-04-10 02:55:08,112 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85364 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-10 02:55:08,112 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/C is initiating minor compaction (all files) 2023-04-10 02:55:08,112 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/C in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:08,113 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/eae585f77f8a448c8c0c54efbac11d99, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/c26733909493467e95f6c235b696dcd5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0614b876287244478bd88c55b895cb36, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/b43bff405a354a5eb6b5e75971f7e08d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/3ef771c4bb984206b377e12d3b7b9dd6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/26c47941beb34ee9b728dc68e0b3d9ff, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/476c0e23750a4aeea31910c6744224c2] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=83.4 K 2023-04-10 02:55:08,113 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 8745bb7edb7040d4b24def654ccc05ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1681095304806 2023-04-10 02:55:08,114 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 4f3579f2c8e64be7988d194d46c1dae5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681095304922 2023-04-10 02:55:08,114 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 5e2c953b053148d4a80fab6b65e8126d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681095306071 2023-04-10 02:55:08,115 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting f4c557dbec0f4d1389bb88e6f2629008, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1681095307192 2023-04-10 02:55:08,115 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting eae585f77f8a448c8c0c54efbac11d99, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681095303653 2023-04-10 02:55:08,116 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting c26733909493467e95f6c235b696dcd5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681095303666 2023-04-10 02:55:08,116 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 0614b876287244478bd88c55b895cb36, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681095304788 2023-04-10 02:55:08,117 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting b43bff405a354a5eb6b5e75971f7e08d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1681095304806 2023-04-10 02:55:08,117 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 3ef771c4bb984206b377e12d3b7b9dd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681095304922 2023-04-10 02:55:08,118 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 26c47941beb34ee9b728dc68e0b3d9ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681095306071 2023-04-10 02:55:08,118 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 476c0e23750a4aeea31910c6744224c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1681095307192 2023-04-10 02:55:08,131 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#A#compaction#269 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:08,138 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#C#compaction#270 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:08,149 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/c34a2a45a9d041b8999705661f59b0c8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/c34a2a45a9d041b8999705661f59b0c8 2023-04-10 02:55:08,170 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/A of de5c1c1f03450ada9cd6207f5cfca947 into c34a2a45a9d041b8999705661f59b0c8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:08,171 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:08,171 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/A, priority=11, startTime=1681095308106; duration=0sec 2023-04-10 02:55:08,172 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:08,172 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:55:08,174 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61394 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:55:08,174 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/B is initiating minor compaction (all files) 2023-04-10 02:55:08,174 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/B in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:08,174 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/5389ca2900d24009b78f42e000b428f3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/25ebf710be6246b495956ba2b2c3303d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/d2758ac3ffed43fc847d74076898a662, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/997c814577ee4482b6bc756071a36196, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a276c6007d8044d8bff5f90c75b8acb4] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=60.0 K 2023-04-10 02:55:08,174 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 5389ca2900d24009b78f42e000b428f3, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681095304788 2023-04-10 02:55:08,175 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 25ebf710be6246b495956ba2b2c3303d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1681095304806 2023-04-10 02:55:08,175 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d2758ac3ffed43fc847d74076898a662, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681095304922 2023-04-10 02:55:08,175 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 997c814577ee4482b6bc756071a36196, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681095306071 2023-04-10 02:55:08,176 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting a276c6007d8044d8bff5f90c75b8acb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1681095307192 2023-04-10 02:55:08,181 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/0085f76de3c74e83b40f3c4667c57936 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0085f76de3c74e83b40f3c4667c57936 2023-04-10 02:55:08,187 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/C of de5c1c1f03450ada9cd6207f5cfca947 into 0085f76de3c74e83b40f3c4667c57936(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:08,187 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:08,187 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/C, priority=9, startTime=1681095308106; duration=0sec 2023-04-10 02:55:08,187 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:08,189 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#B#compaction#271 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:08,191 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:08,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:08,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:08,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:08,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:08,196 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:08,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:08,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:08,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:08,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:08,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:08,198 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,199 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,199 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:08,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:08,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:08,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:08,212 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:08,212 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:08,212 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:08,212 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:08,212 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-10 02:55:08,213 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:08,213 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:08,213 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:08,213 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:08,213 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:08,213 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:08,219 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/ec0b57752d9a4108a266e4245e0a0dc0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/ec0b57752d9a4108a266e4245e0a0dc0 2023-04-10 02:55:08,223 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/B of de5c1c1f03450ada9cd6207f5cfca947 into ec0b57752d9a4108a266e4245e0a0dc0(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:08,223 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:08,223 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/B, priority=11, startTime=1681095308106; duration=0sec 2023-04-10 02:55:08,223 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:08,226 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/13860d9913774b69ae78ea469e26d72f 2023-04-10 02:55:08,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:08,352 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:08,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:08,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095368360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:08,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095368363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:08,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40390 deadline: 1681095368382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:08,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1681095368385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:08,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40414 deadline: 1681095368387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:08,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095368464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:08,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095368467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,642 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/0e6646a4077e441ea9dbe38619571b24 2023-04-10 02:55:08,651 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/d8752cb9168048cd8aae6e9893eafdc9 2023-04-10 02:55:08,655 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/13860d9913774b69ae78ea469e26d72f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/13860d9913774b69ae78ea469e26d72f 2023-04-10 02:55:08,658 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/13860d9913774b69ae78ea469e26d72f, entries=150, sequenceid=361, filesize=11.9 K 2023-04-10 02:55:08,659 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/0e6646a4077e441ea9dbe38619571b24 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/0e6646a4077e441ea9dbe38619571b24 2023-04-10 02:55:08,662 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/0e6646a4077e441ea9dbe38619571b24, entries=150, sequenceid=361, filesize=11.9 K 2023-04-10 02:55:08,663 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/d8752cb9168048cd8aae6e9893eafdc9 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/d8752cb9168048cd8aae6e9893eafdc9 2023-04-10 02:55:08,666 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/d8752cb9168048cd8aae6e9893eafdc9, entries=150, sequenceid=361, filesize=11.9 K 2023-04-10 02:55:08,667 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for de5c1c1f03450ada9cd6207f5cfca947 in 455ms, sequenceid=361, compaction requested=false 2023-04-10 02:55:08,667 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:08,667 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:08,667 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:08,667 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:08,667 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,667 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:08,667 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:08,669 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,669 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:08,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:08,669 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,669 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:08,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:08,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:08,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:08,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:08,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57fa531[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:08,670 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:08,671 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,673 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:08,673 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:55:08,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:08,673 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:08,673 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:08,673 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:08,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:08,673 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:08,674 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:08,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:08,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:08,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:08,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:08,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:08,675 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:08,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:08,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:08,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:08,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:08,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:08,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:08,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:08,679 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:08,679 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:08,679 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:08,679 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:08,679 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:08,681 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,681 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:08,681 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:08,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:08,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:08,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:08,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:08,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:08,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:08,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/a1cc09761aed4f8a8b62498ae28ac831 2023-04-10 02:55:08,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,686 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:08,692 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:08,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,692 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,692 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,692 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,692 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:08,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:08,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:08,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:08,692 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,692 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,692 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:08,692 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:08,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:08,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:08,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:08,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095368707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:08,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095368708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,792 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:08,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:08,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:08,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:08,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:08,796 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:08,796 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:08,796 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:08,796 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:08,796 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,796 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,797 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:08,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:08,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:08,798 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,798 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:08,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:08,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:08,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:08,799 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:08,799 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:08,799 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:08,799 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:08,799 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:08,799 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:08,799 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:08,799 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:08,799 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:08,800 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:08,800 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:08,800 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,800 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:08,800 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:08,801 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:08,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:08,801 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,802 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:08,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:08,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:08,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:08,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:08,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d30b247[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:08,803 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:08,803 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:08,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:08,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:08,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:08,806 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:08,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:08,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:08,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:08,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:08,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:08,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095368811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:08,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:08,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:08,813 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:08,813 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:08,813 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:08,813 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:08,813 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:08,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:08,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095368812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,815 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,815 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:08,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,815 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:08,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:08,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:08,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:08,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:08,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,816 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:08,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,817 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:08,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,819 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,819 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,819 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,819 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,819 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:08,819 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:08,819 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:08,819 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:08,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:08,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:08,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:08,820 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,820 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:08,820 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:08,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:08,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:08,920 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:08,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:08,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:08,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:08,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:08,923 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:08,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:08,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:08,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:08,924 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,924 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,925 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:08,925 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:08,925 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,925 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:08,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:08,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:08,926 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:08,926 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:08,926 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:08,926 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:08,927 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:08,927 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:08,927 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:08,927 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:08,927 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:08,927 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:08,927 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:08,927 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,927 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:08,927 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:08,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,929 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:08,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:08,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:08,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59ca7f19[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:08,931 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:08,931 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,932 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,932 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:08,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:08,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:08,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:08,933 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:08,933 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:08,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,933 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:08,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:08,935 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:08,935 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:08,935 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:08,935 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:08,936 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:08,937 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,937 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:08,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,937 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:08,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:08,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:08,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:08,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:08,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,938 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:08,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,939 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:08,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:08,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,941 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,941 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,941 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:08,941 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:08,941 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:08,941 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:08,941 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:08,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:08,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:08,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:08,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:08,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:08,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:08,942 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:08,942 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:08,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:08,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:09,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:09,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095369014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:09,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095369015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,042 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:09,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:09,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:09,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:09,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:09,048 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:09,048 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:09,048 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:09,048 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:09,049 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,049 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,050 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,050 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:09,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:09,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:09,050 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,050 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:09,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:09,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:09,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:09,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:09,052 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:09,052 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:09,052 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:09,052 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:09,052 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:09,052 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:09,053 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:09,053 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:09,053 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:09,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:09,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:09,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:09,053 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:09,054 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,054 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:09,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:09,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:09,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:09,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:09,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:09,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dc4a6b0[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:09,056 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:09,056 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,058 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,058 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:09,058 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,058 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:09,058 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:09,058 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:09,060 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,060 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:09,060 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:09,060 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:09,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:09,063 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:09,063 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:09,063 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:09,063 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:09,063 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:09,065 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,065 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,065 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:09,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:09,065 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:09,066 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,068 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,069 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,069 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,069 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,069 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,069 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,069 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,069 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:09,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:09,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:09,069 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:09,069 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:09,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:09,069 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:09,069 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:09,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:09,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:09,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:09,097 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/a484f20b540d49d4b0d9ee0c8a7fd396 2023-04-10 02:55:09,169 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:09,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:09,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:09,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:09,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:09,172 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:09,173 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:09,173 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:09,173 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:09,173 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,173 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,178 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:09,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:09,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:09,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:09,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:09,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:09,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:09,179 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:09,179 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:09,179 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:09,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:09,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:09,180 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:09,180 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:09,180 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:09,180 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:09,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:09,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:09,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:09,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:09,182 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,182 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:09,182 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,182 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:09,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:09,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15eac014[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:09,184 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:09,184 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,185 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,185 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:09,185 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:09,185 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:09,185 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,185 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:09,186 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,186 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:09,186 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:09,186 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:09,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:09,189 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:09,189 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:09,189 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:09,189 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:09,190 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:09,192 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,192 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:09,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,192 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:09,192 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,194 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:09,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,196 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,196 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,196 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,196 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,196 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:09,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:09,196 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:09,196 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:09,196 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:09,196 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:09,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:09,196 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:09,197 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:09,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:09,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:09,297 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:09,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:09,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:09,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:09,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:09,300 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:09,300 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:09,300 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:09,300 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:09,300 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,301 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,302 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,302 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:09,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:09,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:09,302 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,302 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:09,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:09,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:09,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:09,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:09,303 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:09,303 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:09,303 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:09,303 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:09,303 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:09,303 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:09,303 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:09,303 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:09,304 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:09,304 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:09,304 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:09,304 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,304 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:09,304 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:09,305 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,305 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:09,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:09,306 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,306 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:09,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:09,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:09,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:09,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:09,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d70fa64[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:09,307 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:09,307 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,309 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,309 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:09,309 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:09,309 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:09,309 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,310 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:09,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:09,311 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:09,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:09,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:09,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:09,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:09,314 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:09,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:09,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:09,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:09,314 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:09,315 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,315 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,315 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:09,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,317 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:09,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:09,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,318 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:09,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:09,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095369317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:09,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095369319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,320 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:09,320 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:09,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:09,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:09,320 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:09,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:09,320 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:09,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:09,320 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:09,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:09,321 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:09,321 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:09,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:09,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40390 deadline: 1681095369390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:09,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40414 deadline: 1681095369392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:09,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1681095369397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,421 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:09,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:09,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:09,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:09,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:09,424 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:09,424 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:09,424 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:09,424 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:09,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:09,427 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:09,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:09,427 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,427 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:09,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:09,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:09,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:09,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:09,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:09,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:09,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:09,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:09,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:09,429 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:09,429 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:09,429 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:09,429 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:09,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:09,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:09,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:09,430 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:09,431 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,431 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:09,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:09,431 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:09,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:09,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:09,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:09,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27c86a64[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:09,433 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:09,433 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,434 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,434 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:09,435 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:09,435 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:09,435 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,435 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:09,436 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:09,436 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:09,436 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,436 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:09,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:09,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:09,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:09,438 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:09,438 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:09,438 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:09,438 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:09,439 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:09,441 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,441 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:09,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,441 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:09,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,443 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:09,446 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,446 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,446 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:09,446 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:09,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:09,446 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:09,446 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,446 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:09,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:09,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:09,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:09,446 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,447 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:09,447 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,447 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:09,447 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:09,507 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/6194772db80248f79790a8b66008e349 2023-04-10 02:55:09,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/a1cc09761aed4f8a8b62498ae28ac831 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/a1cc09761aed4f8a8b62498ae28ac831 2023-04-10 02:55:09,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/a1cc09761aed4f8a8b62498ae28ac831, entries=150, sequenceid=374, filesize=11.9 K 2023-04-10 02:55:09,517 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/a484f20b540d49d4b0d9ee0c8a7fd396 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a484f20b540d49d4b0d9ee0c8a7fd396 2023-04-10 02:55:09,521 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a484f20b540d49d4b0d9ee0c8a7fd396, entries=150, sequenceid=374, filesize=11.9 K 2023-04-10 02:55:09,522 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/6194772db80248f79790a8b66008e349 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/6194772db80248f79790a8b66008e349 2023-04-10 02:55:09,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/6194772db80248f79790a8b66008e349, entries=150, sequenceid=374, filesize=11.9 K 2023-04-10 02:55:09,527 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for de5c1c1f03450ada9cd6207f5cfca947 in 854ms, sequenceid=374, compaction requested=true 2023-04-10 02:55:09,527 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:09,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:09,527 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:09,527 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:09,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:09,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:09,528 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:09,528 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/A is initiating minor compaction (all files) 2023-04-10 02:55:09,529 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/A in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:09,529 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/c34a2a45a9d041b8999705661f59b0c8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/13860d9913774b69ae78ea469e26d72f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/a1cc09761aed4f8a8b62498ae28ac831] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=36.5 K 2023-04-10 02:55:09,529 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:09,529 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/B is initiating minor compaction (all files) 2023-04-10 02:55:09,529 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/B in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:09,529 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/ec0b57752d9a4108a266e4245e0a0dc0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/0e6646a4077e441ea9dbe38619571b24, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a484f20b540d49d4b0d9ee0c8a7fd396] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=36.5 K 2023-04-10 02:55:09,529 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting ec0b57752d9a4108a266e4245e0a0dc0, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1681095307192 2023-04-10 02:55:09,529 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting c34a2a45a9d041b8999705661f59b0c8, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1681095307192 2023-04-10 02:55:09,530 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 0e6646a4077e441ea9dbe38619571b24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1681095307252 2023-04-10 02:55:09,530 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 13860d9913774b69ae78ea469e26d72f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1681095307252 2023-04-10 02:55:09,530 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting a484f20b540d49d4b0d9ee0c8a7fd396, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681095308358 2023-04-10 02:55:09,530 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting a1cc09761aed4f8a8b62498ae28ac831, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681095308358 2023-04-10 02:55:09,538 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#A#compaction#279 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:09,538 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#B#compaction#278 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:09,547 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:09,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:09,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:09,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:09,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:09,552 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:09,552 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:09,552 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:09,552 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:09,553 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,553 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,561 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:09,561 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:09,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:09,561 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/57b86b9a746240c6a367efe8735792be as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/57b86b9a746240c6a367efe8735792be 2023-04-10 02:55:09,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,562 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,562 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:09,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:09,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:09,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:09,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:09,563 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:09,563 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:09,563 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:09,563 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:09,563 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:09,564 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:09,564 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:09,564 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-10 02:55:09,565 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:09,565 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:09,565 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:09,565 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:09,565 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:09,565 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:09,566 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/97c6e31515334260815b481d2a4267c5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/97c6e31515334260815b481d2a4267c5 2023-04-10 02:55:09,567 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/A of de5c1c1f03450ada9cd6207f5cfca947 into 57b86b9a746240c6a367efe8735792be(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:09,567 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:09,567 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/A, priority=13, startTime=1681095309527; duration=0sec 2023-04-10 02:55:09,567 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:09,567 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:09,568 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:09,568 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/C is initiating minor compaction (all files) 2023-04-10 02:55:09,568 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/C in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:09,568 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0085f76de3c74e83b40f3c4667c57936, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/d8752cb9168048cd8aae6e9893eafdc9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/6194772db80248f79790a8b66008e349] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=36.4 K 2023-04-10 02:55:09,572 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 0085f76de3c74e83b40f3c4667c57936, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1681095307192 2023-04-10 02:55:09,574 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting d8752cb9168048cd8aae6e9893eafdc9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1681095307252 2023-04-10 02:55:09,575 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 6194772db80248f79790a8b66008e349, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681095308358 2023-04-10 02:55:09,576 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/B of de5c1c1f03450ada9cd6207f5cfca947 into 97c6e31515334260815b481d2a4267c5(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:09,576 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:09,576 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/B, priority=13, startTime=1681095309527; duration=0sec 2023-04-10 02:55:09,576 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:09,582 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#C#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:09,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:09,823 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:09,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:09,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095369834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:09,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095369835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:09,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095369937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:09,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095369937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:09,983 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/ab2a21b0f59a41a18aa31cc69465acf6 2023-04-10 02:55:10,002 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/66aa8826c8014213a531a5888eac8494 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/66aa8826c8014213a531a5888eac8494 2023-04-10 02:55:10,006 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/C of de5c1c1f03450ada9cd6207f5cfca947 into 66aa8826c8014213a531a5888eac8494(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:10,007 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:10,007 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/C, priority=13, startTime=1681095309527; duration=0sec 2023-04-10 02:55:10,007 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:10,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:10,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095370140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:10,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095370140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,393 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/133363a9037c4c83b480a1ec6e4c1531 2023-04-10 02:55:10,410 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/4d99993af367487e8a9cca64f8c84219 2023-04-10 02:55:10,413 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/ab2a21b0f59a41a18aa31cc69465acf6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/ab2a21b0f59a41a18aa31cc69465acf6 2023-04-10 02:55:10,417 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/ab2a21b0f59a41a18aa31cc69465acf6, entries=150, sequenceid=394, filesize=11.9 K 2023-04-10 02:55:10,418 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/133363a9037c4c83b480a1ec6e4c1531 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/133363a9037c4c83b480a1ec6e4c1531 2023-04-10 02:55:10,421 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/133363a9037c4c83b480a1ec6e4c1531, entries=150, sequenceid=394, filesize=11.9 K 2023-04-10 02:55:10,422 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/4d99993af367487e8a9cca64f8c84219 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/4d99993af367487e8a9cca64f8c84219 2023-04-10 02:55:10,426 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/4d99993af367487e8a9cca64f8c84219, entries=150, sequenceid=394, filesize=11.9 K 2023-04-10 02:55:10,427 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for de5c1c1f03450ada9cd6207f5cfca947 in 863ms, sequenceid=394, compaction requested=false 2023-04-10 02:55:10,427 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:10,427 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:10,427 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:10,427 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:10,428 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,428 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:10,428 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:10,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:10,444 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,444 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,444 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-10 02:55:10,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:10,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:10,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:10,445 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:10,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:10,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:10,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:10,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:10,445 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:10,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:10,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:10,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:10,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:10,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff1b90b[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:10,448 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:10,448 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,449 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,449 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:10,449 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:10,449 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:10,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:10,450 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:10,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:10,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:10,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:10,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:10,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:10,453 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:10,453 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:10,454 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:10,454 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:10,454 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:10,455 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,455 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:10,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,455 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:10,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,456 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:10,457 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/de5b3e97c19d4b83a39d949415965741 2023-04-10 02:55:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,458 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,464 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:10,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,464 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:10,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:10,465 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:10,465 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:10,466 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:10,466 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:10,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:10,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:10,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/01ffcfeb8bc84f8fa2bca77bae35260b 2023-04-10 02:55:10,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:10,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095370473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:10,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095370473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,565 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:10,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:10,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:10,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:10,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:10,568 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:10,568 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:10,568 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:10,568 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:10,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,570 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:10,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:10,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:10,570 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,570 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,571 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:10,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:10,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:10,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095370576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:10,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095370576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:10,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:10,580 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:10,580 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:10,580 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:10,581 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:10,581 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:10,581 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:10,581 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:10,581 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:10,581 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:10,581 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:10,581 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:10,581 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,587 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:10,588 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:10,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:10,590 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:10,591 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,591 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:10,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:10,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:10,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:10,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd82e81[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:10,594 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:10,594 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,595 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,595 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:10,595 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:10,595 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:10,596 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,596 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:10,598 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:10,598 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:10,598 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:10,598 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:10,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:10,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:10,602 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:10,602 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:10,602 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:10,602 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:10,602 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:10,604 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,604 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:10,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,604 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:10,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:10,605 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:10,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,611 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,611 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,611 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,611 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,611 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,611 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,611 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,611 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:10,611 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:10,611 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:10,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:10,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:10,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:10,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:10,612 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:10,612 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:10,712 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:10,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:10,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:10,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:10,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:10,716 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:10,716 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:10,716 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:10,716 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:10,716 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,716 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,717 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:10,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:10,718 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,718 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,719 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:10,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:10,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:10,719 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:10,719 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:10,719 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:10,720 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:10,720 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:10,720 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:10,720 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:10,720 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:10,720 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:10,720 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:10,720 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:10,720 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,720 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:10,720 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:10,723 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:10,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:10,724 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,725 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:10,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:10,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65152a4[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:10,726 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:10,726 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,728 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,728 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:10,728 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:10,728 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:10,729 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,729 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:10,730 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,730 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:10,730 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:10,730 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:10,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:10,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:10,733 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:10,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:10,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:10,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:10,733 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:10,736 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:10,736 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,736 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:10,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:10,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,737 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:10,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,744 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:10,744 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,744 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:10,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:10,744 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:10,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:10,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:10,744 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,744 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,744 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,744 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,744 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:10,744 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,744 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:10,744 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:10,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:10,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095370779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:10,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095370779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,844 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:10,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:10,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:10,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:10,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:10,847 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:10,847 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:10,848 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:10,848 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:10,848 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,848 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,849 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,849 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:10,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:10,850 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,850 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:10,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:10,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:10,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:10,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:10,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:10,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:10,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:10,851 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:10,851 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:10,851 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:10,851 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:10,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:10,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:10,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,852 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:10,852 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:10,853 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,853 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:10,854 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,854 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:10,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:10,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62ae8340[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:10,855 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:10,855 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,856 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,856 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:10,856 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,856 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:10,856 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:10,856 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:10,861 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,861 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:10,861 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:10,861 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:10,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:10,863 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:10,863 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:10,863 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:10,863 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:10,863 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:10,865 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,865 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:10,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:10,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:10,865 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,866 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:10,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:10,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:10,867 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:10,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:10,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:10,868 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:10,869 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:10,869 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:10,869 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:10,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:10,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:10,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:10,869 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,869 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,869 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,869 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:10,869 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:10,869 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:10,869 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:10,869 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:10,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:10,921 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/9d326eca131d46efae354b06b03b2a19 2023-04-10 02:55:10,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/de5b3e97c19d4b83a39d949415965741 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/de5b3e97c19d4b83a39d949415965741 2023-04-10 02:55:10,931 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/de5b3e97c19d4b83a39d949415965741, entries=150, sequenceid=411, filesize=11.9 K 2023-04-10 02:55:10,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/01ffcfeb8bc84f8fa2bca77bae35260b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/01ffcfeb8bc84f8fa2bca77bae35260b 2023-04-10 02:55:10,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/01ffcfeb8bc84f8fa2bca77bae35260b, entries=150, sequenceid=411, filesize=11.9 K 2023-04-10 02:55:10,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/9d326eca131d46efae354b06b03b2a19 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9d326eca131d46efae354b06b03b2a19 2023-04-10 02:55:10,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9d326eca131d46efae354b06b03b2a19, entries=150, sequenceid=411, filesize=11.9 K 2023-04-10 02:55:10,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for de5c1c1f03450ada9cd6207f5cfca947 in 496ms, sequenceid=411, compaction requested=true 2023-04-10 02:55:10,940 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:10,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:10,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:10,940 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:10,940 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:10,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:10,941 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:10,941 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:10,941 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/A is initiating minor compaction (all files) 2023-04-10 02:55:10,941 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/B is initiating minor compaction (all files) 2023-04-10 02:55:10,941 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/A in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:10,941 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/B in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:10,941 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/57b86b9a746240c6a367efe8735792be, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/ab2a21b0f59a41a18aa31cc69465acf6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/de5b3e97c19d4b83a39d949415965741] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=36.6 K 2023-04-10 02:55:10,941 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/97c6e31515334260815b481d2a4267c5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/133363a9037c4c83b480a1ec6e4c1531, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/01ffcfeb8bc84f8fa2bca77bae35260b] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=36.6 K 2023-04-10 02:55:10,942 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 57b86b9a746240c6a367efe8735792be, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681095308358 2023-04-10 02:55:10,942 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 97c6e31515334260815b481d2a4267c5, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681095308358 2023-04-10 02:55:10,942 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ab2a21b0f59a41a18aa31cc69465acf6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681095308700 2023-04-10 02:55:10,942 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 133363a9037c4c83b480a1ec6e4c1531, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681095308700 2023-04-10 02:55:10,943 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting de5b3e97c19d4b83a39d949415965741, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1681095309834 2023-04-10 02:55:10,943 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 01ffcfeb8bc84f8fa2bca77bae35260b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1681095309834 2023-04-10 02:55:10,949 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#B#compaction#288 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:10,949 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#A#compaction#287 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:10,969 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/c3a77cda4c114ae3b0775039276eac84 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/c3a77cda4c114ae3b0775039276eac84 2023-04-10 02:55:10,969 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:10,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:10,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:10,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:10,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:10,974 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:10,974 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:10,974 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:10,974 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:10,975 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/A of de5c1c1f03450ada9cd6207f5cfca947 into c3a77cda4c114ae3b0775039276eac84(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:10,975 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:10,975 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/A, priority=13, startTime=1681095310940; duration=0sec 2023-04-10 02:55:10,975 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:10,975 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:10,976 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,976 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,976 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:10,977 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): de5c1c1f03450ada9cd6207f5cfca947/C is initiating minor compaction (all files) 2023-04-10 02:55:10,977 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de5c1c1f03450ada9cd6207f5cfca947/C in TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:10,977 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/66aa8826c8014213a531a5888eac8494, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/4d99993af367487e8a9cca64f8c84219, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9d326eca131d46efae354b06b03b2a19] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp, totalSize=36.5 K 2023-04-10 02:55:10,977 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 66aa8826c8014213a531a5888eac8494, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681095308358 2023-04-10 02:55:10,978 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 4d99993af367487e8a9cca64f8c84219, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681095308700 2023-04-10 02:55:10,978 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 9d326eca131d46efae354b06b03b2a19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1681095309834 2023-04-10 02:55:10,979 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:10,979 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,979 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:10,979 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:10,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:10,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:10,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:10,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:10,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:10,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:10,981 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:10,981 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:10,981 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:10,981 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:10,981 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:10,982 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:10,982 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:10,982 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-10 02:55:10,982 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:10,982 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:10,982 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:10,982 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:10,982 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:10,982 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:10,983 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): de5c1c1f03450ada9cd6207f5cfca947#C#compaction#289 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:11,000 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/d780c0af18594c4485c423977101a782 2023-04-10 02:55:11,004 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/0ee2dc6d56be4cd5961fede93a85af8e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0ee2dc6d56be4cd5961fede93a85af8e 2023-04-10 02:55:11,009 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/C of de5c1c1f03450ada9cd6207f5cfca947 into 0ee2dc6d56be4cd5961fede93a85af8e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:11,009 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:11,009 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/C, priority=13, startTime=1681095310940; duration=0sec 2023-04-10 02:55:11,009 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:11,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:11,085 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:11,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:11,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095371104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:11,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095371104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:11,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095371207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:11,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095371207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,362 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/6687e12db40044539e013d68c2b93922 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/6687e12db40044539e013d68c2b93922 2023-04-10 02:55:11,367 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de5c1c1f03450ada9cd6207f5cfca947/B of de5c1c1f03450ada9cd6207f5cfca947 into 6687e12db40044539e013d68c2b93922(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:11,367 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:11,367 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., storeName=de5c1c1f03450ada9cd6207f5cfca947/B, priority=13, startTime=1681095310940; duration=0sec 2023-04-10 02:55:11,367 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:11,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:11,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:11,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40414 deadline: 1681095371405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40390 deadline: 1681095371405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,408 DEBUG [Thread-1610] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:55:11,408 DEBUG [Thread-1608] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:55:11,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:11,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095371410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:11,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095371411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,412 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/22b6cb8d16944111bad6ccce893e79c0 2023-04-10 02:55:11,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:11,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1681095371416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,419 DEBUG [Thread-1606] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:55:11,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:11,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095371713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:11,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095371715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,824 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/92bd0f799ae245dc89cfc419fa3b1f26 2023-04-10 02:55:11,828 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/d780c0af18594c4485c423977101a782 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/d780c0af18594c4485c423977101a782 2023-04-10 02:55:11,831 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/d780c0af18594c4485c423977101a782, entries=150, sequenceid=430, filesize=11.9 K 2023-04-10 02:55:11,832 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/22b6cb8d16944111bad6ccce893e79c0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/22b6cb8d16944111bad6ccce893e79c0 2023-04-10 02:55:11,836 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/22b6cb8d16944111bad6ccce893e79c0, entries=150, sequenceid=430, filesize=11.9 K 2023-04-10 02:55:11,836 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/92bd0f799ae245dc89cfc419fa3b1f26 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/92bd0f799ae245dc89cfc419fa3b1f26 2023-04-10 02:55:11,839 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/92bd0f799ae245dc89cfc419fa3b1f26, entries=150, sequenceid=430, filesize=11.9 K 2023-04-10 02:55:11,840 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for de5c1c1f03450ada9cd6207f5cfca947 in 858ms, sequenceid=430, compaction requested=false 2023-04-10 02:55:11,840 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:11,840 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:11,840 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:11,840 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:11,840 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,840 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:11,840 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:11,842 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:11,842 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:11,842 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:11,842 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:11,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:11,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f557ff[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:11,844 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:11,844 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:11,845 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:11,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:11,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:11,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:11,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:11,845 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,845 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:11,846 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:11,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:11,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:11,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:11,849 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:11,849 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:11,849 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:11,849 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:11,849 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:11,851 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:11,851 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:11,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:11,851 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:11,852 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:11,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:11,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:11,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:11,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:11,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:11,853 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:11,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:11,855 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,855 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:11,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:11,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:11,855 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:11,855 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,855 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:11,855 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:11,855 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:11,855 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:11,855 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:11,855 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:11,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:11,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:11,856 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:11,856 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:11,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:11,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:11,956 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:11,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:11,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:11,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:11,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:11,959 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:11,959 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:11,959 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:11,959 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:11,960 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:11,960 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:11,961 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:11,961 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:11,961 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:11,961 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:11,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:11,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:11,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:11,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:11,962 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:11,962 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:11,962 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:11,963 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:11,963 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:11,963 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:11,963 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:11,963 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-10 02:55:11,963 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:11,963 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:11,963 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:11,963 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:11,963 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:11,963 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:12,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:12,220 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:12,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:12,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095372238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:12,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:12,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095372240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:12,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:12,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095372341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:12,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:12,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095372343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:12,371 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/b13f8e3870fa47209278501d2c69c66f 2023-04-10 02:55:12,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:12,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095372543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:12,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:12,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095372546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:12,789 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/62c6f88cfdfa4bbcaff8d788f8b6dec0 2023-04-10 02:55:12,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:12,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095372846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:12,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:12,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095372850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,199 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/973c550f7cce4d3f8e59d4c8ac76df85 2023-04-10 02:55:13,203 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/b13f8e3870fa47209278501d2c69c66f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/b13f8e3870fa47209278501d2c69c66f 2023-04-10 02:55:13,207 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/b13f8e3870fa47209278501d2c69c66f, entries=150, sequenceid=447, filesize=11.9 K 2023-04-10 02:55:13,207 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/62c6f88cfdfa4bbcaff8d788f8b6dec0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/62c6f88cfdfa4bbcaff8d788f8b6dec0 2023-04-10 02:55:13,211 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/62c6f88cfdfa4bbcaff8d788f8b6dec0, entries=150, sequenceid=447, filesize=11.9 K 2023-04-10 02:55:13,211 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/973c550f7cce4d3f8e59d4c8ac76df85 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/973c550f7cce4d3f8e59d4c8ac76df85 2023-04-10 02:55:13,214 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/973c550f7cce4d3f8e59d4c8ac76df85, entries=150, sequenceid=447, filesize=11.9 K 2023-04-10 02:55:13,215 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for de5c1c1f03450ada9cd6207f5cfca947 in 1252ms, sequenceid=447, compaction requested=true 2023-04-10 02:55:13,215 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:13,215 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:13,215 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:13,215 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:13,215 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,215 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:13,215 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:13,217 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:13,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:13,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:13,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:13,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6db4dd46[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:13,219 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:13,219 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:13,220 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:13,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:13,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:13,220 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:13,220 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:13,220 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:13,221 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,221 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:13,223 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:13,223 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:13,223 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,223 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:13,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:13,225 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:13,225 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:13,225 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:13,225 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:13,225 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:13,227 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:13,227 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:13,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:13,227 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:13,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:13,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:13,228 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:13,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:13,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:13,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,231 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:13,231 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:13,231 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:13,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:13,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:13,231 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:13,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:13,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:13,231 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:13,232 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,232 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:13,232 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:13,232 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:13,232 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:13,232 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:13,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:13,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:13,332 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:13,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:13,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:13,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:13,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:13,335 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:13,335 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:13,336 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:13,336 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:13,336 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:13,336 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:13,337 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,337 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:13,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:13,337 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,338 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:13,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:13,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:13,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:13,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:13,339 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:13,339 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:13,339 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:13,339 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:13,339 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:13,339 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:13,339 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:13,339 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-10 02:55:13,339 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:13,339 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:13,339 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:13,339 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:13,339 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:13,339 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:13,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:13,354 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:13,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:13,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095373371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:13,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095373372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:13,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095373475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:13,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095373475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:13,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095373678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:13,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095373679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,747 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/7726e9907edc4d00b60216ea8dbada52 2023-04-10 02:55:13,756 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/90c017a6758c465097b8e7d79633f783 2023-04-10 02:55:13,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:13,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095373983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:13,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:13,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095373983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,173 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/ddc4d6896dbe46fcae022c727beb7ef4 2023-04-10 02:55:14,181 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/7726e9907edc4d00b60216ea8dbada52 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/7726e9907edc4d00b60216ea8dbada52 2023-04-10 02:55:14,184 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/7726e9907edc4d00b60216ea8dbada52, entries=150, sequenceid=464, filesize=11.9 K 2023-04-10 02:55:14,185 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/90c017a6758c465097b8e7d79633f783 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/90c017a6758c465097b8e7d79633f783 2023-04-10 02:55:14,189 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/90c017a6758c465097b8e7d79633f783, entries=150, sequenceid=464, filesize=11.9 K 2023-04-10 02:55:14,190 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/ddc4d6896dbe46fcae022c727beb7ef4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/ddc4d6896dbe46fcae022c727beb7ef4 2023-04-10 02:55:14,193 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/ddc4d6896dbe46fcae022c727beb7ef4, entries=150, sequenceid=464, filesize=11.9 K 2023-04-10 02:55:14,194 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for de5c1c1f03450ada9cd6207f5cfca947 in 855ms, sequenceid=464, compaction requested=true 2023-04-10 02:55:14,194 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:14,194 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:14,197 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:14,197 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:14,197 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:14,197 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61fc7626[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:14,198 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:14,199 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:14,199 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:14,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:14,200 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,200 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:14,201 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:14,201 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:14,201 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,201 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:14,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:14,203 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:14,203 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:14,203 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:14,204 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:14,204 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:14,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:14,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:14,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:14,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:14,209 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:14,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:14,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:14,210 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,212 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,212 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:14,212 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:14,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:14,212 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:14,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:14,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:14,212 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:14,212 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:14,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:14,212 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,212 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:14,212 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:14,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:14,212 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:14,213 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:14,212 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:14,313 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:14,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:14,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:14,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:14,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:14,321 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:14,321 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:14,321 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:14,321 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:14,321 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:14,321 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:14,322 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,323 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:14,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:14,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:14,323 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:14,323 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:14,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:14,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:14,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:14,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:14,324 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:14,324 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:14,324 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:14,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:14,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:14,328 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:14,328 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. started... 2023-04-10 02:55:14,329 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-10 02:55:14,329 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:14,329 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:14,329 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:14,329 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:14,329 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:14,329 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:14,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:14,489 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. as already flushing 2023-04-10 02:55:14,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:14,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095374515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:14,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095374518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:14,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095374620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:14,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095374622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,742 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/54f40d4ee3994736b86833433334212b 2023-04-10 02:55:14,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:14,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095374823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:14,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:14,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095374828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:15,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40424 deadline: 1681095375129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:15,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40360 deadline: 1681095375132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,153 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/acef09490b89412db5412f3ee1b0ab73 2023-04-10 02:55:15,169 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/b4a15adc20694ccb93c22fb843e5248e 2023-04-10 02:55:15,172 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/54f40d4ee3994736b86833433334212b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/54f40d4ee3994736b86833433334212b 2023-04-10 02:55:15,175 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/54f40d4ee3994736b86833433334212b, entries=150, sequenceid=479, filesize=11.9 K 2023-04-10 02:55:15,176 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/acef09490b89412db5412f3ee1b0ab73 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/acef09490b89412db5412f3ee1b0ab73 2023-04-10 02:55:15,179 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/acef09490b89412db5412f3ee1b0ab73, entries=150, sequenceid=479, filesize=11.9 K 2023-04-10 02:55:15,180 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/b4a15adc20694ccb93c22fb843e5248e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/b4a15adc20694ccb93c22fb843e5248e 2023-04-10 02:55:15,183 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/b4a15adc20694ccb93c22fb843e5248e, entries=150, sequenceid=479, filesize=11.9 K 2023-04-10 02:55:15,184 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for de5c1c1f03450ada9cd6207f5cfca947 in 855ms, sequenceid=479, compaction requested=true 2023-04-10 02:55:15,184 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:15,184 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:15,184 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:15,184 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:15,184 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,184 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:15,184 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:15,187 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,187 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:15,187 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:15,187 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:15,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:15,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@775e4dda[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:15,188 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:15,189 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:15,190 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:15,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:15,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:15,190 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:15,190 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:15,190 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:15,190 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,190 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:15,191 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:15,191 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:15,191 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,191 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:15,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:15,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:15,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:15,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:15,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:15,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:15,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:15,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:15,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:15,193 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:15,194 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:15,194 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:15,194 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:15,194 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:15,195 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:15,195 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:15,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:15,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:15,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:15,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:15,195 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:15,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:15,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:15,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:15,197 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:15,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:15,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,201 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:15,201 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:15,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:15,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:15,201 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:15,201 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:15,201 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:15,201 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:15,201 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,201 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:15,201 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:15,201 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:15,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:15,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:15,201 DEBUG [Thread-1612] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:15,201 DEBUG [Thread-1612] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:15,217 DEBUG [Thread-1615] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x753156b2 to 127.0.0.1:60568 2023-04-10 02:55:15,217 DEBUG [Thread-1615] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:15,219 DEBUG [Thread-1621] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c43dddd to 127.0.0.1:60568 2023-04-10 02:55:15,219 DEBUG [Thread-1621] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:15,219 DEBUG [Thread-1619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:60568 2023-04-10 02:55:15,219 DEBUG [Thread-1613] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x104dbf4e to 127.0.0.1:60568 2023-04-10 02:55:15,219 DEBUG [Thread-1619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:15,220 DEBUG [Thread-1613] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:15,221 DEBUG [Thread-1617] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:60568 2023-04-10 02:55:15,221 DEBUG [Thread-1617] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:15,301 DEBUG [Thread-1612] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:15,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:15,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:15,437 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-10 02:55:15,437 DEBUG [Thread-1608] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:60568 2023-04-10 02:55:15,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:15,438 DEBUG [Thread-1608] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:15,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:15,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:15,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:15,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:15,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:15,446 DEBUG [Thread-1610] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x330b77da to 127.0.0.1:60568 2023-04-10 02:55:15,446 DEBUG [Thread-1610] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:15,454 DEBUG [Thread-1606] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:60568 2023-04-10 02:55:15,454 DEBUG [Thread-1606] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:15,633 DEBUG [Thread-1602] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ed1dde9 to 127.0.0.1:60568 2023-04-10 02:55:15,633 DEBUG [Thread-1602] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:15,639 DEBUG [Thread-1604] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70043cf0 to 127.0.0.1:60568 2023-04-10 02:55:15,639 DEBUG [Thread-1604] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 107 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 109 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 55 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 64 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 5101 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 5135 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 5007 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 5107 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 5148 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-10 02:55:15,639 INFO [Listener at localhost/44533] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-10 02:55:15,639 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x112921f2 to 127.0.0.1:60568 2023-04-10 02:55:15,639 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:15,640 INFO [Listener at localhost/44533] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-10 02:55:15,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-10 02:55:15,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:15,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-10 02:55:15,643 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095315643"}]},"ts":"1681095315643"} 2023-04-10 02:55:15,644 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-10 02:55:15,646 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-10 02:55:15,646 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=de5c1c1f03450ada9cd6207f5cfca947, UNASSIGN}] 2023-04-10 02:55:15,648 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=de5c1c1f03450ada9cd6207f5cfca947, UNASSIGN 2023-04-10 02:55:15,648 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=de5c1c1f03450ada9cd6207f5cfca947, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:15,648 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095315648"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095315648"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095315648"}]},"ts":"1681095315648"} 2023-04-10 02:55:15,649 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:55:15,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-10 02:55:15,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:15,801 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing de5c1c1f03450ada9cd6207f5cfca947, disabling compactions & flushes 2023-04-10 02:55:15,801 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:15,844 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/aed1da34b6b64a339ee9a92b9278d95d 2023-04-10 02:55:15,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-10 02:55:16,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-10 02:55:16,253 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/f303b56f51124cb98387a64f573b60ef 2023-04-10 02:55:16,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/e82b825751fe4e588789c8d6975bc4ce 2023-04-10 02:55:16,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/aed1da34b6b64a339ee9a92b9278d95d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/aed1da34b6b64a339ee9a92b9278d95d 2023-04-10 02:55:16,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/aed1da34b6b64a339ee9a92b9278d95d, entries=150, sequenceid=497, filesize=11.9 K 2023-04-10 02:55:16,668 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/f303b56f51124cb98387a64f573b60ef as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/f303b56f51124cb98387a64f573b60ef 2023-04-10 02:55:16,671 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/f303b56f51124cb98387a64f573b60ef, entries=150, sequenceid=497, filesize=11.9 K 2023-04-10 02:55:16,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/e82b825751fe4e588789c8d6975bc4ce as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/e82b825751fe4e588789c8d6975bc4ce 2023-04-10 02:55:16,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/e82b825751fe4e588789c8d6975bc4ce, entries=150, sequenceid=497, filesize=11.9 K 2023-04-10 02:55:16,676 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=26.84 KB/27480 for de5c1c1f03450ada9cd6207f5cfca947 in 1239ms, sequenceid=497, compaction requested=true 2023-04-10 02:55:16,676 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:16,676 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:16,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:16,676 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. because compaction request was cancelled 2023-04-10 02:55:16,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:16,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:16,676 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. because compaction request was cancelled 2023-04-10 02:55:16,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. after waiting 0 ms 2023-04-10 02:55:16,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:16,676 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. because compaction request was cancelled 2023-04-10 02:55:16,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:16,676 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing de5c1c1f03450ada9cd6207f5cfca947 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-10 02:55:16,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=A 2023-04-10 02:55:16,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:16,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=B 2023-04-10 02:55:16,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:16,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de5c1c1f03450ada9cd6207f5cfca947, store=C 2023-04-10 02:55:16,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:16,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-10 02:55:17,082 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/66b3d5bf7cd04c41a5db58c05f6cba5b 2023-04-10 02:55:17,491 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/e546f6b92bc94e2d963a46cbfcd6d0ed 2023-04-10 02:55:17,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-10 02:55:17,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/72bcaf35ad58436c9d7fcf6860033045 2023-04-10 02:55:17,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/A/66b3d5bf7cd04c41a5db58c05f6cba5b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/66b3d5bf7cd04c41a5db58c05f6cba5b 2023-04-10 02:55:17,905 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/66b3d5bf7cd04c41a5db58c05f6cba5b, entries=100, sequenceid=504, filesize=9.5 K 2023-04-10 02:55:17,906 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/B/e546f6b92bc94e2d963a46cbfcd6d0ed as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/e546f6b92bc94e2d963a46cbfcd6d0ed 2023-04-10 02:55:17,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/e546f6b92bc94e2d963a46cbfcd6d0ed, entries=100, sequenceid=504, filesize=9.5 K 2023-04-10 02:55:17,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/.tmp/C/72bcaf35ad58436c9d7fcf6860033045 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/72bcaf35ad58436c9d7fcf6860033045 2023-04-10 02:55:17,914 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/72bcaf35ad58436c9d7fcf6860033045, entries=100, sequenceid=504, filesize=9.5 K 2023-04-10 02:55:17,914 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for de5c1c1f03450ada9cd6207f5cfca947 in 1238ms, sequenceid=504, compaction requested=true 2023-04-10 02:55:17,936 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/39981446dd5d4a6c8147daf8a60356cb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/7d3887e3e1ce420dab862b9701b1159a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/41898603d8f84aba8079ca83c3a8cc8b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/89d02c2a11f144d0bb9c15a63c63f140, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/9e10cce422ba4712bdde1e0700166643, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/1960aae77cdb4c0482b6abc56cd353b2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/64cd512df88e4a6f945e98da43c18e44, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/0a7bc07c08a047f886f1a8497f03a7ba, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/737aa86776b543979f9fcf4591cc03d4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/62c20e6d39994490a3cc512fe27e1a24, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/3b7775afd48e4943941fab02b7e82caa, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/540b3a73324844b0afd1d77c5ed3247c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/1bc7797a2c4d402ba5dc34037f63cab4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/fa8ee734ecb144c695a3b1d570ffdbda, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/845efcfc8edc45c3a8194f0e9752fcf1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/721734ef1ce348b69b84ffecb11e88f2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/fd347fcead4947209674a49ba3b0f679, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/38d5595578f241009da2d4747b4bd0a8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/11445e9958a2444ead4f83441c91f3a8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/8745bb7edb7040d4b24def654ccc05ea, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/4f3579f2c8e64be7988d194d46c1dae5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/5e2c953b053148d4a80fab6b65e8126d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/c34a2a45a9d041b8999705661f59b0c8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/f4c557dbec0f4d1389bb88e6f2629008, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/13860d9913774b69ae78ea469e26d72f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/57b86b9a746240c6a367efe8735792be, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/a1cc09761aed4f8a8b62498ae28ac831, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/ab2a21b0f59a41a18aa31cc69465acf6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/de5b3e97c19d4b83a39d949415965741] to archive 2023-04-10 02:55:17,937 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:55:17,939 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/39981446dd5d4a6c8147daf8a60356cb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/39981446dd5d4a6c8147daf8a60356cb 2023-04-10 02:55:17,940 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/7d3887e3e1ce420dab862b9701b1159a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/7d3887e3e1ce420dab862b9701b1159a 2023-04-10 02:55:17,941 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/41898603d8f84aba8079ca83c3a8cc8b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/41898603d8f84aba8079ca83c3a8cc8b 2023-04-10 02:55:17,942 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/89d02c2a11f144d0bb9c15a63c63f140 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/89d02c2a11f144d0bb9c15a63c63f140 2023-04-10 02:55:17,942 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/9e10cce422ba4712bdde1e0700166643 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/9e10cce422ba4712bdde1e0700166643 2023-04-10 02:55:17,943 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/1960aae77cdb4c0482b6abc56cd353b2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/1960aae77cdb4c0482b6abc56cd353b2 2023-04-10 02:55:17,944 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/64cd512df88e4a6f945e98da43c18e44 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/64cd512df88e4a6f945e98da43c18e44 2023-04-10 02:55:17,945 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/0a7bc07c08a047f886f1a8497f03a7ba to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/0a7bc07c08a047f886f1a8497f03a7ba 2023-04-10 02:55:17,945 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/737aa86776b543979f9fcf4591cc03d4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/737aa86776b543979f9fcf4591cc03d4 2023-04-10 02:55:17,946 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/62c20e6d39994490a3cc512fe27e1a24 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/62c20e6d39994490a3cc512fe27e1a24 2023-04-10 02:55:17,947 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/3b7775afd48e4943941fab02b7e82caa to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/3b7775afd48e4943941fab02b7e82caa 2023-04-10 02:55:17,948 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/540b3a73324844b0afd1d77c5ed3247c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/540b3a73324844b0afd1d77c5ed3247c 2023-04-10 02:55:17,948 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/1bc7797a2c4d402ba5dc34037f63cab4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/1bc7797a2c4d402ba5dc34037f63cab4 2023-04-10 02:55:17,949 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/fa8ee734ecb144c695a3b1d570ffdbda to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/fa8ee734ecb144c695a3b1d570ffdbda 2023-04-10 02:55:17,950 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/845efcfc8edc45c3a8194f0e9752fcf1 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/845efcfc8edc45c3a8194f0e9752fcf1 2023-04-10 02:55:17,951 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/721734ef1ce348b69b84ffecb11e88f2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/721734ef1ce348b69b84ffecb11e88f2 2023-04-10 02:55:17,951 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/fd347fcead4947209674a49ba3b0f679 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/fd347fcead4947209674a49ba3b0f679 2023-04-10 02:55:17,952 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/38d5595578f241009da2d4747b4bd0a8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/38d5595578f241009da2d4747b4bd0a8 2023-04-10 02:55:17,953 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/11445e9958a2444ead4f83441c91f3a8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/11445e9958a2444ead4f83441c91f3a8 2023-04-10 02:55:17,954 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/8745bb7edb7040d4b24def654ccc05ea to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/8745bb7edb7040d4b24def654ccc05ea 2023-04-10 02:55:17,955 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/4f3579f2c8e64be7988d194d46c1dae5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/4f3579f2c8e64be7988d194d46c1dae5 2023-04-10 02:55:17,956 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/5e2c953b053148d4a80fab6b65e8126d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/5e2c953b053148d4a80fab6b65e8126d 2023-04-10 02:55:17,956 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/c34a2a45a9d041b8999705661f59b0c8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/c34a2a45a9d041b8999705661f59b0c8 2023-04-10 02:55:17,957 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/f4c557dbec0f4d1389bb88e6f2629008 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/f4c557dbec0f4d1389bb88e6f2629008 2023-04-10 02:55:17,958 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/13860d9913774b69ae78ea469e26d72f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/13860d9913774b69ae78ea469e26d72f 2023-04-10 02:55:17,959 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/57b86b9a746240c6a367efe8735792be to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/57b86b9a746240c6a367efe8735792be 2023-04-10 02:55:17,960 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/a1cc09761aed4f8a8b62498ae28ac831 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/a1cc09761aed4f8a8b62498ae28ac831 2023-04-10 02:55:17,961 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/ab2a21b0f59a41a18aa31cc69465acf6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/ab2a21b0f59a41a18aa31cc69465acf6 2023-04-10 02:55:17,961 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/de5b3e97c19d4b83a39d949415965741 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/de5b3e97c19d4b83a39d949415965741 2023-04-10 02:55:17,966 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/07dc44ae6e974c97827f567ff2ac555a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9cd25fcb2f6645ad8a5020d65fd3e405, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/2e5276274f4344ff8d26f36cbf257613, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/c9cf2fea1fac4bdea1b790957c22f977, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/e7421710c0b9455388a655b91ba68c9f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/78545b45e3254f46b6b139577470783b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/b0edc22730cf4ce99fa8ad45c93cb087, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/2b14e2e5cbb748319870059b3c7cc625, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9de45e5f092a46c1a9eed75f6e6596d8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/33bfa16911a74c33a4f3d7e88621f208, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/1f1427f1147b4db394e6488062fdcc2a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a533c09634734945825bf4fc4076a7e3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/3b80eeac39c345678ac3be15d88f199d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/f4f4921fc68f477492233eb530f27bfa, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/54c12902a1864049a362f85ea3bd722a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/8b78ac654ad446e5aec132835bb3442b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/6ae99ed40ada4eb8aba6852d33312217, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/5389ca2900d24009b78f42e000b428f3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9791942f97e249cdb2c726dc1ab4682e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/25ebf710be6246b495956ba2b2c3303d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/d2758ac3ffed43fc847d74076898a662, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/997c814577ee4482b6bc756071a36196, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/ec0b57752d9a4108a266e4245e0a0dc0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a276c6007d8044d8bff5f90c75b8acb4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/0e6646a4077e441ea9dbe38619571b24, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/97c6e31515334260815b481d2a4267c5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a484f20b540d49d4b0d9ee0c8a7fd396, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/133363a9037c4c83b480a1ec6e4c1531, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/01ffcfeb8bc84f8fa2bca77bae35260b] to archive 2023-04-10 02:55:17,966 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:55:17,967 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/07dc44ae6e974c97827f567ff2ac555a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/07dc44ae6e974c97827f567ff2ac555a 2023-04-10 02:55:17,968 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9cd25fcb2f6645ad8a5020d65fd3e405 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9cd25fcb2f6645ad8a5020d65fd3e405 2023-04-10 02:55:17,969 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/2e5276274f4344ff8d26f36cbf257613 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/2e5276274f4344ff8d26f36cbf257613 2023-04-10 02:55:17,970 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/c9cf2fea1fac4bdea1b790957c22f977 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/c9cf2fea1fac4bdea1b790957c22f977 2023-04-10 02:55:17,971 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/e7421710c0b9455388a655b91ba68c9f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/e7421710c0b9455388a655b91ba68c9f 2023-04-10 02:55:17,972 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/78545b45e3254f46b6b139577470783b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/78545b45e3254f46b6b139577470783b 2023-04-10 02:55:17,973 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/b0edc22730cf4ce99fa8ad45c93cb087 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/b0edc22730cf4ce99fa8ad45c93cb087 2023-04-10 02:55:17,974 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/2b14e2e5cbb748319870059b3c7cc625 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/2b14e2e5cbb748319870059b3c7cc625 2023-04-10 02:55:17,975 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9de45e5f092a46c1a9eed75f6e6596d8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9de45e5f092a46c1a9eed75f6e6596d8 2023-04-10 02:55:17,976 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/33bfa16911a74c33a4f3d7e88621f208 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/33bfa16911a74c33a4f3d7e88621f208 2023-04-10 02:55:17,976 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/1f1427f1147b4db394e6488062fdcc2a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/1f1427f1147b4db394e6488062fdcc2a 2023-04-10 02:55:17,977 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a533c09634734945825bf4fc4076a7e3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a533c09634734945825bf4fc4076a7e3 2023-04-10 02:55:17,978 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/3b80eeac39c345678ac3be15d88f199d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/3b80eeac39c345678ac3be15d88f199d 2023-04-10 02:55:17,979 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/f4f4921fc68f477492233eb530f27bfa to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/f4f4921fc68f477492233eb530f27bfa 2023-04-10 02:55:17,980 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/54c12902a1864049a362f85ea3bd722a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/54c12902a1864049a362f85ea3bd722a 2023-04-10 02:55:17,980 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/8b78ac654ad446e5aec132835bb3442b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/8b78ac654ad446e5aec132835bb3442b 2023-04-10 02:55:17,981 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/6ae99ed40ada4eb8aba6852d33312217 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/6ae99ed40ada4eb8aba6852d33312217 2023-04-10 02:55:17,982 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/5389ca2900d24009b78f42e000b428f3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/5389ca2900d24009b78f42e000b428f3 2023-04-10 02:55:17,983 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9791942f97e249cdb2c726dc1ab4682e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/9791942f97e249cdb2c726dc1ab4682e 2023-04-10 02:55:17,983 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/25ebf710be6246b495956ba2b2c3303d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/25ebf710be6246b495956ba2b2c3303d 2023-04-10 02:55:17,984 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/d2758ac3ffed43fc847d74076898a662 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/d2758ac3ffed43fc847d74076898a662 2023-04-10 02:55:17,985 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/997c814577ee4482b6bc756071a36196 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/997c814577ee4482b6bc756071a36196 2023-04-10 02:55:17,986 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/ec0b57752d9a4108a266e4245e0a0dc0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/ec0b57752d9a4108a266e4245e0a0dc0 2023-04-10 02:55:17,987 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a276c6007d8044d8bff5f90c75b8acb4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a276c6007d8044d8bff5f90c75b8acb4 2023-04-10 02:55:17,987 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/0e6646a4077e441ea9dbe38619571b24 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/0e6646a4077e441ea9dbe38619571b24 2023-04-10 02:55:17,988 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/97c6e31515334260815b481d2a4267c5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/97c6e31515334260815b481d2a4267c5 2023-04-10 02:55:17,989 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a484f20b540d49d4b0d9ee0c8a7fd396 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/a484f20b540d49d4b0d9ee0c8a7fd396 2023-04-10 02:55:17,989 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/133363a9037c4c83b480a1ec6e4c1531 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/133363a9037c4c83b480a1ec6e4c1531 2023-04-10 02:55:17,990 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/01ffcfeb8bc84f8fa2bca77bae35260b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/01ffcfeb8bc84f8fa2bca77bae35260b 2023-04-10 02:55:17,993 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/61ec709206f14c1eaea11a6c978e7087, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/65fa94c563004511a490a5bce9f50908, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/56548abb001a4042a81b2fc956b0050a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/62cc740b7e624036a0bbd9eeacf49dc4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/d5fce69634e9409f887f7720d9e626df, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9b8ad5e4e6ff493599c1fb91ebd25ee9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/6515e6271a7f4b3eb3bdac25a58ea3a9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9a104d69a28b4bb097dfec30ac2c1b43, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/5f5fccf3164945d7835edf4e24607247, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/668d42727eb84e1b8cbb9f45acaf63c4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/50b1a57123b544a78adce8fd8f501471, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/02739fce8b7d4cdd8b3525962fc4f9e9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/cc231a18bfb44a1c8bb61c527688f271, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/be27b1c874414f1186da1c314d2bda04, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/eae585f77f8a448c8c0c54efbac11d99, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/95adb0e52d184ab8b05d417193a05992, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/c26733909493467e95f6c235b696dcd5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0614b876287244478bd88c55b895cb36, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/b43bff405a354a5eb6b5e75971f7e08d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/3ef771c4bb984206b377e12d3b7b9dd6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/26c47941beb34ee9b728dc68e0b3d9ff, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0085f76de3c74e83b40f3c4667c57936, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/476c0e23750a4aeea31910c6744224c2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/d8752cb9168048cd8aae6e9893eafdc9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/66aa8826c8014213a531a5888eac8494, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/6194772db80248f79790a8b66008e349, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/4d99993af367487e8a9cca64f8c84219, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9d326eca131d46efae354b06b03b2a19] to archive 2023-04-10 02:55:17,994 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:55:17,995 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/61ec709206f14c1eaea11a6c978e7087 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/61ec709206f14c1eaea11a6c978e7087 2023-04-10 02:55:17,995 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/65fa94c563004511a490a5bce9f50908 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/65fa94c563004511a490a5bce9f50908 2023-04-10 02:55:17,996 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/56548abb001a4042a81b2fc956b0050a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/56548abb001a4042a81b2fc956b0050a 2023-04-10 02:55:17,997 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/62cc740b7e624036a0bbd9eeacf49dc4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/62cc740b7e624036a0bbd9eeacf49dc4 2023-04-10 02:55:17,998 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/d5fce69634e9409f887f7720d9e626df to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/d5fce69634e9409f887f7720d9e626df 2023-04-10 02:55:17,998 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9b8ad5e4e6ff493599c1fb91ebd25ee9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9b8ad5e4e6ff493599c1fb91ebd25ee9 2023-04-10 02:55:17,999 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/6515e6271a7f4b3eb3bdac25a58ea3a9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/6515e6271a7f4b3eb3bdac25a58ea3a9 2023-04-10 02:55:18,000 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9a104d69a28b4bb097dfec30ac2c1b43 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9a104d69a28b4bb097dfec30ac2c1b43 2023-04-10 02:55:18,000 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/5f5fccf3164945d7835edf4e24607247 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/5f5fccf3164945d7835edf4e24607247 2023-04-10 02:55:18,001 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/668d42727eb84e1b8cbb9f45acaf63c4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/668d42727eb84e1b8cbb9f45acaf63c4 2023-04-10 02:55:18,002 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/50b1a57123b544a78adce8fd8f501471 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/50b1a57123b544a78adce8fd8f501471 2023-04-10 02:55:18,002 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/02739fce8b7d4cdd8b3525962fc4f9e9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/02739fce8b7d4cdd8b3525962fc4f9e9 2023-04-10 02:55:18,003 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/cc231a18bfb44a1c8bb61c527688f271 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/cc231a18bfb44a1c8bb61c527688f271 2023-04-10 02:55:18,004 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/be27b1c874414f1186da1c314d2bda04 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/be27b1c874414f1186da1c314d2bda04 2023-04-10 02:55:18,005 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/eae585f77f8a448c8c0c54efbac11d99 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/eae585f77f8a448c8c0c54efbac11d99 2023-04-10 02:55:18,006 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/95adb0e52d184ab8b05d417193a05992 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/95adb0e52d184ab8b05d417193a05992 2023-04-10 02:55:18,007 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/c26733909493467e95f6c235b696dcd5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/c26733909493467e95f6c235b696dcd5 2023-04-10 02:55:18,008 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0614b876287244478bd88c55b895cb36 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0614b876287244478bd88c55b895cb36 2023-04-10 02:55:18,008 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/b43bff405a354a5eb6b5e75971f7e08d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/b43bff405a354a5eb6b5e75971f7e08d 2023-04-10 02:55:18,009 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/3ef771c4bb984206b377e12d3b7b9dd6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/3ef771c4bb984206b377e12d3b7b9dd6 2023-04-10 02:55:18,010 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/26c47941beb34ee9b728dc68e0b3d9ff to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/26c47941beb34ee9b728dc68e0b3d9ff 2023-04-10 02:55:18,011 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0085f76de3c74e83b40f3c4667c57936 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0085f76de3c74e83b40f3c4667c57936 2023-04-10 02:55:18,012 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/476c0e23750a4aeea31910c6744224c2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/476c0e23750a4aeea31910c6744224c2 2023-04-10 02:55:18,013 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/d8752cb9168048cd8aae6e9893eafdc9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/d8752cb9168048cd8aae6e9893eafdc9 2023-04-10 02:55:18,013 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/66aa8826c8014213a531a5888eac8494 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/66aa8826c8014213a531a5888eac8494 2023-04-10 02:55:18,014 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/6194772db80248f79790a8b66008e349 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/6194772db80248f79790a8b66008e349 2023-04-10 02:55:18,015 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/4d99993af367487e8a9cca64f8c84219 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/4d99993af367487e8a9cca64f8c84219 2023-04-10 02:55:18,016 DEBUG [StoreCloser-TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9d326eca131d46efae354b06b03b2a19 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/9d326eca131d46efae354b06b03b2a19 2023-04-10 02:55:18,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/recovered.edits/507.seqid, newMaxSeqId=507, maxSeqId=1 2023-04-10 02:55:18,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947. 2023-04-10 02:55:18,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for de5c1c1f03450ada9cd6207f5cfca947: 2023-04-10 02:55:18,022 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:18,022 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=de5c1c1f03450ada9cd6207f5cfca947, regionState=CLOSED 2023-04-10 02:55:18,022 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095318022"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095318022"}]},"ts":"1681095318022"} 2023-04-10 02:55:18,024 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-10 02:55:18,024 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure de5c1c1f03450ada9cd6207f5cfca947, server=jenkins-hbase4.apache.org,38321,1681095222501 in 2.3740 sec 2023-04-10 02:55:18,025 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-10 02:55:18,026 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=de5c1c1f03450ada9cd6207f5cfca947, UNASSIGN in 2.3780 sec 2023-04-10 02:55:18,026 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095318026"}]},"ts":"1681095318026"} 2023-04-10 02:55:18,027 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-10 02:55:18,028 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-10 02:55:18,029 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.3890 sec 2023-04-10 02:55:19,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-10 02:55:19,748 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-10 02:55:19,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-10 02:55:19,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:19,751 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:19,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-10 02:55:19,751 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:19,755 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:19,757 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/recovered.edits] 2023-04-10 02:55:19,759 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/54f40d4ee3994736b86833433334212b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/54f40d4ee3994736b86833433334212b 2023-04-10 02:55:19,760 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/66b3d5bf7cd04c41a5db58c05f6cba5b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/66b3d5bf7cd04c41a5db58c05f6cba5b 2023-04-10 02:55:19,761 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/7726e9907edc4d00b60216ea8dbada52 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/7726e9907edc4d00b60216ea8dbada52 2023-04-10 02:55:19,762 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/aed1da34b6b64a339ee9a92b9278d95d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/aed1da34b6b64a339ee9a92b9278d95d 2023-04-10 02:55:19,763 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/b13f8e3870fa47209278501d2c69c66f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/b13f8e3870fa47209278501d2c69c66f 2023-04-10 02:55:19,763 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/c3a77cda4c114ae3b0775039276eac84 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/c3a77cda4c114ae3b0775039276eac84 2023-04-10 02:55:19,764 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/d780c0af18594c4485c423977101a782 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/A/d780c0af18594c4485c423977101a782 2023-04-10 02:55:19,766 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/22b6cb8d16944111bad6ccce893e79c0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/22b6cb8d16944111bad6ccce893e79c0 2023-04-10 02:55:19,767 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/62c6f88cfdfa4bbcaff8d788f8b6dec0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/62c6f88cfdfa4bbcaff8d788f8b6dec0 2023-04-10 02:55:19,768 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/6687e12db40044539e013d68c2b93922 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/6687e12db40044539e013d68c2b93922 2023-04-10 02:55:19,768 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/90c017a6758c465097b8e7d79633f783 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/90c017a6758c465097b8e7d79633f783 2023-04-10 02:55:19,769 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/acef09490b89412db5412f3ee1b0ab73 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/acef09490b89412db5412f3ee1b0ab73 2023-04-10 02:55:19,770 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/e546f6b92bc94e2d963a46cbfcd6d0ed to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/e546f6b92bc94e2d963a46cbfcd6d0ed 2023-04-10 02:55:19,771 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/f303b56f51124cb98387a64f573b60ef to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/B/f303b56f51124cb98387a64f573b60ef 2023-04-10 02:55:19,773 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0ee2dc6d56be4cd5961fede93a85af8e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/0ee2dc6d56be4cd5961fede93a85af8e 2023-04-10 02:55:19,774 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/72bcaf35ad58436c9d7fcf6860033045 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/72bcaf35ad58436c9d7fcf6860033045 2023-04-10 02:55:19,774 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/92bd0f799ae245dc89cfc419fa3b1f26 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/92bd0f799ae245dc89cfc419fa3b1f26 2023-04-10 02:55:19,775 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/973c550f7cce4d3f8e59d4c8ac76df85 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/973c550f7cce4d3f8e59d4c8ac76df85 2023-04-10 02:55:19,776 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/b4a15adc20694ccb93c22fb843e5248e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/b4a15adc20694ccb93c22fb843e5248e 2023-04-10 02:55:19,777 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/ddc4d6896dbe46fcae022c727beb7ef4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/ddc4d6896dbe46fcae022c727beb7ef4 2023-04-10 02:55:19,778 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/e82b825751fe4e588789c8d6975bc4ce to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/C/e82b825751fe4e588789c8d6975bc4ce 2023-04-10 02:55:19,780 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/recovered.edits/507.seqid to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947/recovered.edits/507.seqid 2023-04-10 02:55:19,780 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/de5c1c1f03450ada9cd6207f5cfca947 2023-04-10 02:55:19,780 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-10 02:55:19,782 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:19,783 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-10 02:55:19,784 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-10 02:55:19,785 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:19,785 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-10 02:55:19,785 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681095319785"}]},"ts":"9223372036854775807"} 2023-04-10 02:55:19,787 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-10 02:55:19,787 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => de5c1c1f03450ada9cd6207f5cfca947, NAME => 'TestAcidGuarantees,,1681095293035.de5c1c1f03450ada9cd6207f5cfca947.', STARTKEY => '', ENDKEY => ''}] 2023-04-10 02:55:19,787 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-10 02:55:19,787 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681095319787"}]},"ts":"9223372036854775807"} 2023-04-10 02:55:19,788 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-10 02:55:19,790 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:19,791 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 41 msec 2023-04-10 02:55:19,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-10 02:55:19,852 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-10 02:55:19,864 INFO [Listener at localhost/44533] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 224) - Thread LEAK? -, OpenFileDescriptor=558 (was 527) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=388 (was 392), ProcessCount=184 (was 184), AvailableMemoryMB=1373 (was 1538) 2023-04-10 02:55:19,875 INFO [Listener at localhost/44533] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=558, MaxFileDescriptor=60000, SystemLoadAverage=388, ProcessCount=184, AvailableMemoryMB=1373 2023-04-10 02:55:19,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-10 02:55:19,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-10 02:55:19,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:19,879 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-10 02:55:19,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-10 02:55:19,880 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-10 02:55:19,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-10 02:55:19,881 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:19,881 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524 empty. 2023-04-10 02:55:19,882 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:19,882 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-10 02:55:19,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-10 02:55:20,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-10 02:55:20,288 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-10 02:55:20,290 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => f91f1c15c884b2146b2b8d0ffac5d524, NAME => 'TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp 2023-04-10 02:55:20,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-10 02:55:20,695 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:55:20,696 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing f91f1c15c884b2146b2b8d0ffac5d524, disabling compactions & flushes 2023-04-10 02:55:20,696 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:20,696 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:20,696 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. after waiting 0 ms 2023-04-10 02:55:20,696 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:20,696 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:20,696 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:20,698 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-10 02:55:20,699 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095320698"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095320698"}]},"ts":"1681095320698"} 2023-04-10 02:55:20,700 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-10 02:55:20,700 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-10 02:55:20,700 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095320700"}]},"ts":"1681095320700"} 2023-04-10 02:55:20,701 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-10 02:55:20,703 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f91f1c15c884b2146b2b8d0ffac5d524, ASSIGN}] 2023-04-10 02:55:20,705 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f91f1c15c884b2146b2b8d0ffac5d524, ASSIGN 2023-04-10 02:55:20,705 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f91f1c15c884b2146b2b8d0ffac5d524, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38321,1681095222501; forceNewPlan=false, retain=false 2023-04-10 02:55:20,856 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=f91f1c15c884b2146b2b8d0ffac5d524, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:20,857 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095320856"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095320856"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095320856"}]},"ts":"1681095320856"} 2023-04-10 02:55:20,858 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:55:20,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-10 02:55:21,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:21,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f91f1c15c884b2146b2b8d0ffac5d524, NAME => 'TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.', STARTKEY => '', ENDKEY => ''} 2023-04-10 02:55:21,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:21,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:55:21,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:21,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:21,016 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:21,018 DEBUG [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A 2023-04-10 02:55:21,018 DEBUG [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A 2023-04-10 02:55:21,018 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:55:21,018 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f91f1c15c884b2146b2b8d0ffac5d524 columnFamilyName A 2023-04-10 02:55:21,019 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.HStore(310): Store=f91f1c15c884b2146b2b8d0ffac5d524/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:55:21,019 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:21,020 DEBUG [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B 2023-04-10 02:55:21,020 DEBUG [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B 2023-04-10 02:55:21,020 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:55:21,020 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f91f1c15c884b2146b2b8d0ffac5d524 columnFamilyName B 2023-04-10 02:55:21,021 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.HStore(310): Store=f91f1c15c884b2146b2b8d0ffac5d524/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:55:21,021 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:21,022 DEBUG [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C 2023-04-10 02:55:21,022 DEBUG [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C 2023-04-10 02:55:21,022 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:55:21,022 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f91f1c15c884b2146b2b8d0ffac5d524 columnFamilyName C 2023-04-10 02:55:21,023 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.HStore(310): Store=f91f1c15c884b2146b2b8d0ffac5d524/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:55:21,023 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:21,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:21,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:21,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-10 02:55:21,027 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:21,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-10 02:55:21,029 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f91f1c15c884b2146b2b8d0ffac5d524; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74691546, jitterRate=0.11299076676368713}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-10 02:55:21,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:21,030 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., pid=37, masterSystemTime=1681095321009 2023-04-10 02:55:21,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:21,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:21,031 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=f91f1c15c884b2146b2b8d0ffac5d524, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:21,032 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095321031"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681095321031"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681095321031"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681095321031"}]},"ts":"1681095321031"} 2023-04-10 02:55:21,034 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-10 02:55:21,034 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 in 175 msec 2023-04-10 02:55:21,035 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-10 02:55:21,035 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f91f1c15c884b2146b2b8d0ffac5d524, ASSIGN in 331 msec 2023-04-10 02:55:21,036 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-10 02:55:21,036 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095321036"}]},"ts":"1681095321036"} 2023-04-10 02:55:21,037 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-10 02:55:21,040 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-10 02:55:21,041 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1630 sec 2023-04-10 02:55:21,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-10 02:55:21,984 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-10 02:55:21,985 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x4473f45c to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:21,990 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fd3d54e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:21,991 DEBUG [Listener at localhost/44533] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:21,993 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:21,994 DEBUG [Listener at localhost/44533] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-10 02:55:21,995 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-10 02:55:21,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-10 02:55:21,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-10 02:55:21,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:22,409 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-10 02:55:22,410 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-10 02:55:22,410 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-10 02:55:22,412 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-10 02:55:22,414 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f91f1c15c884b2146b2b8d0ffac5d524, REOPEN/MOVE}] 2023-04-10 02:55:22,415 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f91f1c15c884b2146b2b8d0ffac5d524, REOPEN/MOVE 2023-04-10 02:55:22,415 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=f91f1c15c884b2146b2b8d0ffac5d524, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,416 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095322415"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095322415"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095322415"}]},"ts":"1681095322415"} 2023-04-10 02:55:22,417 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:55:22,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:22,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f91f1c15c884b2146b2b8d0ffac5d524, disabling compactions & flushes 2023-04-10 02:55:22,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:22,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:22,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. after waiting 0 ms 2023-04-10 02:55:22,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:22,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-10 02:55:22,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:22,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:22,573 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: f91f1c15c884b2146b2b8d0ffac5d524 to self. 2023-04-10 02:55:22,574 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:22,575 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=f91f1c15c884b2146b2b8d0ffac5d524, regionState=CLOSED 2023-04-10 02:55:22,575 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095322575"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095322575"}]},"ts":"1681095322575"} 2023-04-10 02:55:22,577 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-10 02:55:22,577 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 in 159 msec 2023-04-10 02:55:22,577 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f91f1c15c884b2146b2b8d0ffac5d524, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,38321,1681095222501; forceNewPlan=false, retain=true 2023-04-10 02:55:22,727 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=f91f1c15c884b2146b2b8d0ffac5d524, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,728 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095322727"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095322727"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095322727"}]},"ts":"1681095322727"} 2023-04-10 02:55:22,729 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:55:22,883 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:22,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f91f1c15c884b2146b2b8d0ffac5d524, NAME => 'TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.', STARTKEY => '', ENDKEY => ''} 2023-04-10 02:55:22,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:22,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:55:22,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:22,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:22,885 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:22,885 DEBUG [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A 2023-04-10 02:55:22,886 DEBUG [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A 2023-04-10 02:55:22,886 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:55:22,886 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f91f1c15c884b2146b2b8d0ffac5d524 columnFamilyName A 2023-04-10 02:55:22,887 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.HStore(310): Store=f91f1c15c884b2146b2b8d0ffac5d524/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:55:22,888 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:22,888 DEBUG [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B 2023-04-10 02:55:22,888 DEBUG [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B 2023-04-10 02:55:22,888 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:55:22,889 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f91f1c15c884b2146b2b8d0ffac5d524 columnFamilyName B 2023-04-10 02:55:22,889 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.HStore(310): Store=f91f1c15c884b2146b2b8d0ffac5d524/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:55:22,889 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:22,890 DEBUG [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C 2023-04-10 02:55:22,890 DEBUG [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C 2023-04-10 02:55:22,890 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:55:22,890 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f91f1c15c884b2146b2b8d0ffac5d524 columnFamilyName C 2023-04-10 02:55:22,890 INFO [StoreOpener-f91f1c15c884b2146b2b8d0ffac5d524-1] regionserver.HStore(310): Store=f91f1c15c884b2146b2b8d0ffac5d524/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:55:22,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:22,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:22,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:22,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-10 02:55:22,894 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:22,895 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f91f1c15c884b2146b2b8d0ffac5d524; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64805606, jitterRate=-0.034321218729019165}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-10 02:55:22,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:22,895 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., pid=42, masterSystemTime=1681095322880 2023-04-10 02:55:22,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:22,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:22,897 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=f91f1c15c884b2146b2b8d0ffac5d524, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,897 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095322897"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681095322897"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681095322897"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681095322897"}]},"ts":"1681095322897"} 2023-04-10 02:55:22,899 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-10 02:55:22,899 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 in 169 msec 2023-04-10 02:55:22,900 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-10 02:55:22,900 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f91f1c15c884b2146b2b8d0ffac5d524, REOPEN/MOVE in 485 msec 2023-04-10 02:55:22,901 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-10 02:55:22,902 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-10 02:55:22,903 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 905 msec 2023-04-10 02:55:22,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-10 02:55:22,904 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x0bb27fd4 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:22,909 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49b5bd0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:22,909 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x445919fd to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:22,912 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c8d927d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:22,913 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x4d010713 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:22,917 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@419b69bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:22,917 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x69100738 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:22,920 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c6cac96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:22,920 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x6f603c27 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:22,924 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a0df34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:22,924 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x10ee5f06 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:22,928 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10377b50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:22,928 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x21128196 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:22,932 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4366fcdd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:22,932 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e86d9 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:22,938 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a35a28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:22,939 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x711ebaf2 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:22,942 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a1d4246, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:22,942 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x0a7503d4 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:22,945 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@eb8f142, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:22,950 DEBUG [hconnection-0x371fcd58-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:22,950 DEBUG [hconnection-0xbaf3b8d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:22,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:22,951 DEBUG [hconnection-0xc2dc720-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:22,952 DEBUG [hconnection-0x37781939-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:22,952 DEBUG [hconnection-0x4b987f3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:22,952 DEBUG [hconnection-0x2284ca9e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:22,952 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36646, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:22,953 DEBUG [hconnection-0xa2497d3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:22,953 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36666, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:22,954 DEBUG [hconnection-0x607ab6f0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:22,954 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36652, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:22,954 DEBUG [hconnection-0x18c59f3a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:22,956 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36692, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:22,957 DEBUG [hconnection-0xbc50524-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:22,958 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36728, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:22,960 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36650, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:22,960 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36690, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:22,960 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36680, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:22,960 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36712, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:22,960 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36704, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:22,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:22,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:22,970 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:22,970 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:22,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:22,972 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:22,972 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:22,972 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:55:22,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:22,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:22,973 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:22,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:22,973 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:22,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:22,973 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:22,973 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:22,976 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:22,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:22,976 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:22,977 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,977 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:22,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:22,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:22,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:22,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:22,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:22,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:22,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:22,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:22,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:22,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:22,985 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:22,985 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:22,985 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:22,985 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:22,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:22,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:22,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,986 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:22,986 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:22,987 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,987 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:22,987 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:22,987 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:22,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:22,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dbea661[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:22,990 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:22,990 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:22,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:22,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:22,991 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:22,991 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:22,991 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,991 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:22,991 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:22,996 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:22,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095322983, max timestamp=1681095322995Num uniques 859; 2023-04-10 02:55:22,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095322983, max timestamp=1681095322995Num uniques 859; 2023-04-10 02:55:22,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:55:22,997 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:22,997 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:22,997 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:22,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095322983, max timestamp=1681095322993Num uniques 859; 2023-04-10 02:55:22,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:55:22,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:55:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:22,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:22,998 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:22,999 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:22,999 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:22,999 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:22,999 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:23,000 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,000 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,000 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,001 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,002 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:23,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:23,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,004 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,005 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,005 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,005 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:23,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,005 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,005 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,005 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:23,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095383012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095383012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36690 deadline: 1681095383013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36680 deadline: 1681095383013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36666 deadline: 1681095383015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,105 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:23,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:23,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:23,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:23,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:23,111 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:23,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:23,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:23,112 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,112 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,113 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,113 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,113 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,113 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:23,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:23,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:23,116 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:23,116 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,116 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:23,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095383118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095383118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36690 deadline: 1681095383119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36680 deadline: 1681095383119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36666 deadline: 1681095383119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:23,121 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,122 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:23,122 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:23,122 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:23,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:23,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:23,122 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:23,126 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,126 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,126 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,126 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ab0e7d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:23,127 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:23,127 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,128 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:23,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:23,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:23,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:23,130 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,131 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:23,131 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,131 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:23,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:23,132 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:23,133 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:23,133 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:23,133 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:23,133 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:23,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,135 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,136 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,138 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:23,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:23,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,139 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,139 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:23,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,139 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:23,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,239 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:23,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:23,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:23,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:23,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:23,243 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:23,243 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,243 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:23,243 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:23,243 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,243 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,245 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,245 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,245 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,245 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:23,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:23,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:23,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:23,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:23,250 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,251 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:23,251 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:23,251 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:23,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:23,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:23,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:23,253 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,253 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,253 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,253 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:23,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@146417c8[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:23,255 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:23,255 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,256 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,256 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,256 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:23,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,256 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:23,256 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:23,257 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:23,259 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:23,259 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,259 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:23,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:23,261 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:23,261 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:23,261 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:23,261 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:23,261 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:23,263 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,263 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,263 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,263 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,264 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,266 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,267 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:23,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:23,267 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,266 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,267 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,267 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:23,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:23,267 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:23,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36666 deadline: 1681095383321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095383321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095383321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36690 deadline: 1681095383322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36680 deadline: 1681095383322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,367 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:23,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:23,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:23,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:23,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:23,370 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:23,370 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,370 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:23,370 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:23,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,372 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,372 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,373 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,373 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:23,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:23,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:23,374 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,374 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:23,374 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:23,374 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,377 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:23,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d1b803[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:23,378 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:23,378 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,379 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:23,380 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,380 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:23,381 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:23,381 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,381 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:23,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:23,383 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:23,383 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:23,383 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:23,383 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:23,383 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:23,384 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,384 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,384 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,385 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,386 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,391 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304106615c3a4feca438490b92512b73faf25 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304106615c3a4feca438490b92512b73faf25 2023-04-10 02:55:23,392 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,392 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,392 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,392 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,392 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,392 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,392 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:23,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:23,392 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,392 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,392 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,392 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:23,392 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:23,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:23,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,493 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:23,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:23,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:23,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:23,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:23,496 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:23,496 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,496 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:23,496 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:23,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,498 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,499 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,499 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:23,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:23,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:23,501 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,501 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:23,501 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:23,501 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,502 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:23,502 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:23,502 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,502 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:23,502 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:23,504 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,505 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,505 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:23,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@397489f6[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:23,507 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:23,507 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,509 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,510 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:23,510 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:23,510 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:23,510 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,510 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:23,511 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,511 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:23,511 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,511 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:23,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:23,514 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:23,514 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:23,514 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:23,514 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:23,514 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:23,515 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,515 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,516 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,517 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,518 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,521 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,521 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,521 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,521 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,521 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:23,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:23,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,521 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,521 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:23,521 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:23,521 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,521 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,521 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,521 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:23,543 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-10 02:55:23,621 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:23,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:23,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:23,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:23,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:23,625 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:23,625 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,625 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:23,625 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:23,626 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,626 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,628 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,628 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,628 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,628 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:23,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:23,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:23,629 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:23,629 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,629 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:23,629 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:23,629 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:23,629 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,629 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:23,630 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:23,630 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:23,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36666 deadline: 1681095383625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095383625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095383626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,632 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36690 deadline: 1681095383627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:23,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36680 deadline: 1681095383627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:23,633 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:23,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7762a043[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:23,633 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,634 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,635 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:23,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:23,635 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:23,635 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:23,636 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,636 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:23,636 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,636 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:23,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:23,638 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:23,638 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:23,638 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:23,639 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:23,639 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:23,641 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,641 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,641 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,642 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,644 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,646 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,646 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,646 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,646 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:23,646 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,646 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:23,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:23,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,647 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:23,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,647 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:23,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,747 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:23,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:23,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:23,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:23,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:23,751 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:23,751 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,751 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:23,751 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:23,751 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,751 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,752 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,752 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,752 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,753 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:23,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:23,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:23,754 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,754 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:23,754 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:23,754 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:23,756 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,756 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,756 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,756 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:23,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@438c2330[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:23,757 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:23,757 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,760 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:23,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:23,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:23,760 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,760 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:23,761 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,761 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:23,761 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,761 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:23,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:23,765 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:23,765 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:23,765 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:23,765 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:23,765 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:23,767 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,767 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,767 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,767 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,774 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:23,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:23,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,775 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,775 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,775 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,775 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:23,775 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,775 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:23,775 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:23,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,799 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/9d30ac409d314f1d84f10445cefd5b78 2023-04-10 02:55:23,875 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:23,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:23,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:23,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:23,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:23,879 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:23,879 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,879 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:23,879 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:23,879 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,879 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,880 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,880 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,881 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,881 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:23,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:23,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:23,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:23,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:23,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:23,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:23,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:23,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:23,882 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,882 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:23,882 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:23,882 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:23,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:23,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:23,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:23,882 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:23,884 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,884 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:23,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f819ae2[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:23,885 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:23,886 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,887 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:23,887 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,887 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:23,890 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:23,890 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,890 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,890 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:23,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:23,892 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:23,892 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:23,892 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:23,892 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:23,892 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:23,894 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,894 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,894 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:23,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,895 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:23,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,896 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:23,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,898 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:23,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:23,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:23,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:23,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:23,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:23,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:23,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:23,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:23,899 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:23,899 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:23,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,899 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:23,899 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:23,999 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:23,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:24,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:24,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:24,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:24,003 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:24,003 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:24,003 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:24,003 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:24,003 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,003 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:24,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:24,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:24,004 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:24,005 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,005 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:24,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:24,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:24,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:24,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:24,006 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:24,006 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:24,007 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:24,007 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:24,007 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:24,007 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:24,007 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,007 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:24,007 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:24,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,010 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:24,011 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,011 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:24,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:24,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a793dd[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:24,012 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:24,012 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,013 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,013 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:24,013 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:24,013 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:24,013 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,014 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:24,015 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:24,015 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:24,015 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:24,015 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:24,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:24,017 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:24,017 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:24,017 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:24,017 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:24,017 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:24,019 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,019 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:24,019 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:24,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:24,020 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:24,021 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,025 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:24,025 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:24,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:24,025 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:24,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:24,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:24,026 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:24,026 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:24,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:24,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:24,027 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,027 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:24,027 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:24,027 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,027 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,027 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,027 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,126 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:24,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:24,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:24,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:24,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:24,129 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:24,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:24,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:24,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:24,130 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,130 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:24,131 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:24,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:24,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:24,131 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:24,131 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,131 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:24,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:24,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:24,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:24,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:24,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:24,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:24,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:24,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:24,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:24,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:24,133 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:24,133 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:24,133 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:24,133 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:24,133 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:24,133 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:24,133 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,133 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:24,133 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:24,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:24,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:24,135 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:24,135 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:24,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:24,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095384132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:24,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:24,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36666 deadline: 1681095384133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:24,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:24,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a54372[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:24,137 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:24,137 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:24,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36680 deadline: 1681095384134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:24,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095384135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:24,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36690 deadline: 1681095384135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,138 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:24,138 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,138 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:24,140 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:24,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:24,142 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:24,142 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:24,143 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:24,143 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:24,143 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:24,144 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:24,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:24,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:24,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:24,145 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,148 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:24,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:24,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:24,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:24,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:24,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:24,150 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:24,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:24,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,150 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:24,150 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:24,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:24,150 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:24,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/9759e005628b43bb94efd726d43744a3 2023-04-10 02:55:24,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/0d45321fa98c4724b086a676fd37caec 2023-04-10 02:55:24,224 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/9d30ac409d314f1d84f10445cefd5b78 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/9d30ac409d314f1d84f10445cefd5b78 2023-04-10 02:55:24,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/9d30ac409d314f1d84f10445cefd5b78, entries=100, sequenceid=14, filesize=18.4 K 2023-04-10 02:55:24,229 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/9759e005628b43bb94efd726d43744a3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/9759e005628b43bb94efd726d43744a3 2023-04-10 02:55:24,232 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/9759e005628b43bb94efd726d43744a3, entries=100, sequenceid=14, filesize=9.3 K 2023-04-10 02:55:24,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/0d45321fa98c4724b086a676fd37caec as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/0d45321fa98c4724b086a676fd37caec 2023-04-10 02:55:24,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/0d45321fa98c4724b086a676fd37caec, entries=100, sequenceid=14, filesize=9.3 K 2023-04-10 02:55:24,237 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f91f1c15c884b2146b2b8d0ffac5d524 in 1265ms, sequenceid=14, compaction requested=false 2023-04-10 02:55:24,237 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-10 02:55:24,237 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:24,251 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:24,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:24,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:24,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:24,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:24,254 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:24,254 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:24,254 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:24,254 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:24,254 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,254 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:24,259 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:24,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:24,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:24,259 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:24,259 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:24,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:24,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:24,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:24,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:24,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:24,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:24,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:24,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:24,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:24,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:24,261 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:24,261 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:24,261 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-10 02:55:24,261 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:24,261 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:24,261 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:24,261 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:24,262 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:24,262 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:24,279 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410143202de8fc04978a8c97f405aa86a9c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410143202de8fc04978a8c97f405aa86a9c 2023-04-10 02:55:24,684 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/eebf4c95b68548eb9b77cba44468464c 2023-04-10 02:55:25,095 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/97ee68bb77c84805a0cc90af0f214fa3 2023-04-10 02:55:25,106 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/1dc666c6964843b2813e263cb89a742e 2023-04-10 02:55:25,109 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/eebf4c95b68548eb9b77cba44468464c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/eebf4c95b68548eb9b77cba44468464c 2023-04-10 02:55:25,114 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/eebf4c95b68548eb9b77cba44468464c, entries=150, sequenceid=51, filesize=25.1 K 2023-04-10 02:55:25,115 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/97ee68bb77c84805a0cc90af0f214fa3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/97ee68bb77c84805a0cc90af0f214fa3 2023-04-10 02:55:25,119 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/97ee68bb77c84805a0cc90af0f214fa3, entries=150, sequenceid=51, filesize=11.6 K 2023-04-10 02:55:25,120 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/1dc666c6964843b2813e263cb89a742e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/1dc666c6964843b2813e263cb89a742e 2023-04-10 02:55:25,123 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/1dc666c6964843b2813e263cb89a742e, entries=150, sequenceid=51, filesize=11.6 K 2023-04-10 02:55:25,124 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for f91f1c15c884b2146b2b8d0ffac5d524 in 863ms, sequenceid=51, compaction requested=false 2023-04-10 02:55:25,127 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:25,127 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:25,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:25,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:25,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:25,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:25,129 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:25,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21271f5[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:25,131 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:25,131 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,132 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,132 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,133 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:25,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:25,133 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:25,133 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:25,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:25,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:25,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:25,142 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:25,142 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:25,142 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:25,142 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:25,142 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:25,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:25,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,145 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,146 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,149 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,149 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,149 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,149 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,149 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:25,149 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:25,149 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:25,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:25,149 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:25,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:25,149 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:25,149 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:25,149 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:25,149 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:25,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:25,225 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:55:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:25,249 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:25,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:25,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:25,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:25,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:25,255 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:25,255 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:25,255 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:25,255 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:25,256 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,256 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,257 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:25,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:25,257 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,258 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,258 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:25,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:25,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:25,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:25,259 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:25,259 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:25,260 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:25,260 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:25,260 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:25,260 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:25,262 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:25,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5117addf[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:25,264 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:25,264 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,265 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,266 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,266 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:25,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,266 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:25,266 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:25,266 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:25,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095385260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095385261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095325255, max timestamp=1681095325266Num uniques 859; 2023-04-10 02:55:25,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:55:25,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095325255, max timestamp=1681095325266Num uniques 859; 2023-04-10 02:55:25,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:55:25,269 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:25,269 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095325255, max timestamp=1681095325266Num uniques 859; 2023-04-10 02:55:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,269 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:25,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:55:25,269 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:25,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:25,272 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:25,272 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:25,272 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:25,272 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:25,272 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:25,273 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,273 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,273 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:25,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:25,274 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,276 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,277 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,277 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,278 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,278 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,278 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:25,278 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:25,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:25,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:25,278 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:25,278 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:25,278 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:25,278 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:25,278 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:25,278 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:25,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36690 deadline: 1681095385303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36666 deadline: 1681095385309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36680 deadline: 1681095385311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095385369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095385370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,378 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:25,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:25,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:25,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:25,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:25,384 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:25,384 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:25,384 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:25,384 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:25,385 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,385 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,387 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:25,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:25,387 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,388 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,388 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:25,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:25,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:25,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:25,389 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:25,389 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:25,390 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:25,390 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:25,390 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:25,390 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:25,390 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,390 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:25,390 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:25,392 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,392 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,392 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,392 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:25,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f50e55b[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:25,396 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:25,396 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,399 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:25,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:25,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:25,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:25,401 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:25,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:25,407 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:25,407 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:25,407 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:25,407 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:25,407 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:25,411 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,411 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:25,411 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,416 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,417 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,424 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:25,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36690 deadline: 1681095385415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:25,425 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,425 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,425 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,425 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:25,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:25,425 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:25,425 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:25,425 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:25,425 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:25,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36680 deadline: 1681095385418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,426 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:25,426 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36666 deadline: 1681095385419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,525 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:25,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:25,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:25,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:25,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:25,532 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:25,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:25,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:25,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:25,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,533 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,534 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:25,534 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:25,534 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,534 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:25,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:25,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:25,535 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:25,535 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:25,535 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:25,535 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:25,535 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:25,535 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:25,536 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:25,536 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:25,536 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:25,542 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,543 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,543 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:25,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25e48d10[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:25,545 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:25,545 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:25,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:25,547 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:25,547 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:25,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,547 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:25,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:25,550 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:25,550 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:25,550 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:25,550 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:25,550 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:25,552 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,552 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,552 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:25,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,553 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,554 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,559 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,559 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,559 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,559 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,559 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,559 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,559 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:25,560 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:25,560 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:25,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:25,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:25,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:25,560 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:25,560 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:25,560 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:25,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:25,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095385578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095385579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36690 deadline: 1681095385626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36680 deadline: 1681095385628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36666 deadline: 1681095385629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,657 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304100abdef583aa24d63bf8531c5d13a5a48 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304100abdef583aa24d63bf8531c5d13a5a48 2023-04-10 02:55:25,660 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:25,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:25,662 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/5b0a81761ed94076a4f47d24a8889de4 2023-04-10 02:55:25,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:25,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:25,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:25,664 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:25,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:25,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:25,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:25,665 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,665 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,666 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:25,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:25,666 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:25,666 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,666 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:25,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:25,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:25,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:25,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/a3bc55160d154b958ba683b6c874f64c 2023-04-10 02:55:25,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:25,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:25,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:25,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:25,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:25,676 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:25,676 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:25,677 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:25,677 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:25,677 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:25,677 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:25,677 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,677 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:25,677 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:25,678 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,679 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,679 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:25,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e8cd80[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:25,681 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:25,681 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,682 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,682 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:25,682 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:25,682 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:25,683 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,683 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:25,684 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,685 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:25,685 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:25,686 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,687 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/1a89c361285f41d8a3d1fffaa14b6bfb 2023-04-10 02:55:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:25,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:25,689 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:25,689 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:25,689 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:25,689 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:25,689 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:25,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/5b0a81761ed94076a4f47d24a8889de4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/5b0a81761ed94076a4f47d24a8889de4 2023-04-10 02:55:25,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:25,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:25,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:25,692 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:25,694 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/5b0a81761ed94076a4f47d24a8889de4, entries=350, sequenceid=61, filesize=52.3 K 2023-04-10 02:55:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,694 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/a3bc55160d154b958ba683b6c874f64c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/a3bc55160d154b958ba683b6c874f64c 2023-04-10 02:55:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,698 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:25,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:25,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:25,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/a3bc55160d154b958ba683b6c874f64c, entries=150, sequenceid=61, filesize=11.6 K 2023-04-10 02:55:25,698 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,698 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:25,698 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,698 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:25,698 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:25,698 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:25,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:25,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:25,698 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:25,698 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/1a89c361285f41d8a3d1fffaa14b6bfb as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/1a89c361285f41d8a3d1fffaa14b6bfb 2023-04-10 02:55:25,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:25,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/1a89c361285f41d8a3d1fffaa14b6bfb, entries=150, sequenceid=61, filesize=11.6 K 2023-04-10 02:55:25,702 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f91f1c15c884b2146b2b8d0ffac5d524 in 477ms, sequenceid=61, compaction requested=true 2023-04-10 02:55:25,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:25,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:25,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:25,702 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:25,702 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:25,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:25,703 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:25,703 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:25,703 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/A is initiating minor compaction (all files) 2023-04-10 02:55:25,703 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/B is initiating minor compaction (all files) 2023-04-10 02:55:25,703 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/A in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:25,703 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/B in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:25,703 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/9d30ac409d314f1d84f10445cefd5b78, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/eebf4c95b68548eb9b77cba44468464c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/5b0a81761ed94076a4f47d24a8889de4] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=95.8 K 2023-04-10 02:55:25,703 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/9759e005628b43bb94efd726d43744a3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/97ee68bb77c84805a0cc90af0f214fa3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/a3bc55160d154b958ba683b6c874f64c] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=32.5 K 2023-04-10 02:55:25,703 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 9d30ac409d314f1d84f10445cefd5b78, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681095322965 2023-04-10 02:55:25,704 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 9759e005628b43bb94efd726d43744a3, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681095322965 2023-04-10 02:55:25,704 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting eebf4c95b68548eb9b77cba44468464c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681095323012 2023-04-10 02:55:25,704 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 97ee68bb77c84805a0cc90af0f214fa3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681095323012 2023-04-10 02:55:25,704 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 5b0a81761ed94076a4f47d24a8889de4, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681095325184 2023-04-10 02:55:25,704 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting a3bc55160d154b958ba683b6c874f64c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681095325212 2023-04-10 02:55:25,716 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#B#compaction#318 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:25,721 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#A#compaction#317 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:25,798 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:25,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:25,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:25,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:25,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:25,802 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:25,802 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:25,802 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:25,802 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:25,803 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,803 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:25,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:25,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:25,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:25,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:25,806 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:25,806 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:25,806 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:25,806 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:25,806 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:25,806 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:25,806 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:25,806 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-10 02:55:25,806 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:25,806 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:25,807 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:25,807 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:25,807 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:25,807 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:25,825 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304102e55d5dc0ae54f20b4cde16574de7124 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304102e55d5dc0ae54f20b4cde16574de7124 2023-04-10 02:55:25,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:25,890 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:25,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095385921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095385928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36690 deadline: 1681095385931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36680 deadline: 1681095385933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:25,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:25,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36666 deadline: 1681095385934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:26,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095386029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:26,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095386031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,139 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/6ca4706c30244fbe989c279156c73349 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/6ca4706c30244fbe989c279156c73349 2023-04-10 02:55:26,143 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/B of f91f1c15c884b2146b2b8d0ffac5d524 into 6ca4706c30244fbe989c279156c73349(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:26,143 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:26,143 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/B, priority=13, startTime=1681095325702; duration=0sec 2023-04-10 02:55:26,144 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:26,144 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:26,145 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:26,145 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/C is initiating minor compaction (all files) 2023-04-10 02:55:26,145 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/C in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:26,145 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/0d45321fa98c4724b086a676fd37caec, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/1dc666c6964843b2813e263cb89a742e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/1a89c361285f41d8a3d1fffaa14b6bfb] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=32.5 K 2023-04-10 02:55:26,145 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 0d45321fa98c4724b086a676fd37caec, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681095322965 2023-04-10 02:55:26,145 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 1dc666c6964843b2813e263cb89a742e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681095323012 2023-04-10 02:55:26,146 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 1a89c361285f41d8a3d1fffaa14b6bfb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681095325212 2023-04-10 02:55:26,151 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#C#compaction#320 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:26,233 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/543f3193a37d4c64a8dfe76037a3ea52 2023-04-10 02:55:26,244 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/5066d8615b224398bca9f6f950a696f4 2023-04-10 02:55:26,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:26,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095386240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:26,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095386240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:26,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36680 deadline: 1681095386442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:26,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36666 deadline: 1681095386443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:26,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36690 deadline: 1681095386445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:26,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095386547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:26,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095386548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,558 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/3642b6f02e0c40c3b00b9d62fcd4c69e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/3642b6f02e0c40c3b00b9d62fcd4c69e 2023-04-10 02:55:26,561 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/C of f91f1c15c884b2146b2b8d0ffac5d524 into 3642b6f02e0c40c3b00b9d62fcd4c69e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:26,562 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:26,562 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/C, priority=13, startTime=1681095325702; duration=0sec 2023-04-10 02:55:26,562 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:26,655 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/48d3a3ada45a4f7295a01323d6fb7067 2023-04-10 02:55:26,659 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/543f3193a37d4c64a8dfe76037a3ea52 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/543f3193a37d4c64a8dfe76037a3ea52 2023-04-10 02:55:26,662 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/543f3193a37d4c64a8dfe76037a3ea52, entries=150, sequenceid=98, filesize=25.1 K 2023-04-10 02:55:26,663 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/5066d8615b224398bca9f6f950a696f4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/5066d8615b224398bca9f6f950a696f4 2023-04-10 02:55:26,666 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/5066d8615b224398bca9f6f950a696f4, entries=150, sequenceid=98, filesize=11.6 K 2023-04-10 02:55:26,667 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/48d3a3ada45a4f7295a01323d6fb7067 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/48d3a3ada45a4f7295a01323d6fb7067 2023-04-10 02:55:26,673 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/48d3a3ada45a4f7295a01323d6fb7067, entries=150, sequenceid=98, filesize=11.6 K 2023-04-10 02:55:26,674 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for f91f1c15c884b2146b2b8d0ffac5d524 in 868ms, sequenceid=98, compaction requested=false 2023-04-10 02:55:26,674 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:26,674 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:26,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:26,677 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,677 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:26,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:26,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:26,677 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:26,678 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:26,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:26,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:26,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:26,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:26,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:26,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@587df797[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:26,679 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:26,679 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:26,680 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:26,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:26,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:26,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:26,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:26,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:26,680 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,680 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:26,681 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,682 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:26,682 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:26,682 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:26,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:26,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:26,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:26,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:26,684 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:26,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:26,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:26,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:26,684 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:26,686 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:26,686 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:26,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:26,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:26,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:26,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:26,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:26,686 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:26,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:26,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:26,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:26,686 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:26,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:26,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:26,691 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:26,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:26,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:26,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:26,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:26,692 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:26,692 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:26,692 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:26,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:26,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:26,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:26,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:26,792 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:26,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:26,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:26,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:26,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:26,795 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:26,795 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:26,795 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:26,795 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:26,795 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:26,795 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:26,797 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:26,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:26,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:26,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:26,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:26,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:26,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:26,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:26,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:26,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:26,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:26,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:26,798 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:26,798 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:26,798 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:55:26,799 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:26,799 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:26,799 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:26,799 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:26,799 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:26,799 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:26,955 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/347c651f9b5544d29c5b2ca52056036d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/347c651f9b5544d29c5b2ca52056036d 2023-04-10 02:55:26,961 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/A of f91f1c15c884b2146b2b8d0ffac5d524 into 347c651f9b5544d29c5b2ca52056036d(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-10 02:55:26,961 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:26,961 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/A, priority=13, startTime=1681095325702; duration=1sec 2023-04-10 02:55:26,961 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:27,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:27,059 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:27,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095327088, max timestamp=1681095327102Num uniques 859; 2023-04-10 02:55:27,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095327088, max timestamp=1681095327102Num uniques 859; 2023-04-10 02:55:27,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095327088, max timestamp=1681095327102Num uniques 859; 2023-04-10 02:55:27,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:27,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:27,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:27,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:27,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095387159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:27,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:27,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095387160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:27,219 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410fd6986cc3c794cff98080962dde7237b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fd6986cc3c794cff98080962dde7237b 2023-04-10 02:55:27,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:27,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095387267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:27,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:27,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095387267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:27,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36666 deadline: 1681095387452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:27,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36690 deadline: 1681095387453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:27,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36680 deadline: 1681095387455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:27,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:27,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095387481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:27,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:27,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095387482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:27,668 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=110, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/7a8fcf0d20f345dd91a574be55db0ae9 2023-04-10 02:55:27,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:27,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:27,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095387787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:27,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095387788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,082 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/e87d0eff2f574e9fb313fcbc3578904b 2023-04-10 02:55:28,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:28,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095388294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:28,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095388298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,504 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/fe503a2b82d543d2bea92aa077417e85 2023-04-10 02:55:28,509 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/7a8fcf0d20f345dd91a574be55db0ae9 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/7a8fcf0d20f345dd91a574be55db0ae9 2023-04-10 02:55:28,513 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/7a8fcf0d20f345dd91a574be55db0ae9, entries=150, sequenceid=110, filesize=25.1 K 2023-04-10 02:55:28,515 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/e87d0eff2f574e9fb313fcbc3578904b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/e87d0eff2f574e9fb313fcbc3578904b 2023-04-10 02:55:28,520 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/e87d0eff2f574e9fb313fcbc3578904b, entries=150, sequenceid=110, filesize=11.6 K 2023-04-10 02:55:28,521 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/fe503a2b82d543d2bea92aa077417e85 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe503a2b82d543d2bea92aa077417e85 2023-04-10 02:55:28,526 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe503a2b82d543d2bea92aa077417e85, entries=150, sequenceid=110, filesize=11.6 K 2023-04-10 02:55:28,526 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f91f1c15c884b2146b2b8d0ffac5d524 in 1728ms, sequenceid=110, compaction requested=true 2023-04-10 02:55:28,527 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:28,527 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:28,527 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:28,527 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:28,527 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,527 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:28,527 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:28,529 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:28,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:28,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70afbbfe[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:28,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:28,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:28,533 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:28,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:28,533 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:28,533 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:28,533 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:28,533 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,533 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:28,538 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,538 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:28,539 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:28,539 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:28,541 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:28,541 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:28,541 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:28,541 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:28,541 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:28,542 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:28,542 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:28,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:28,542 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:28,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:28,543 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:28,544 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:28,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:28,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:28,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:28,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:28,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:28,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:28,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:28,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:28,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:28,547 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:28,547 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:28,547 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:28,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:28,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:28,647 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:28,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:28,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:28,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:28,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:28,650 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:28,650 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:28,650 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:28,650 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:28,651 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:28,651 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:28,653 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,653 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:28,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:28,653 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:28,653 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:28,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:28,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:28,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:28,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:28,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:28,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:28,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:28,654 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:28,654 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:28,654 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:28,655 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-10 02:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:28,676 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304103937613beebb4fbfb4148cd3ca6e198f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304103937613beebb4fbfb4148cd3ca6e198f 2023-04-10 02:55:29,094 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/85eaaf96e11b4253bc9dcea70de49eab 2023-04-10 02:55:29,104 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/f69e5198878f4887afccc84d08abd2b0 2023-04-10 02:55:29,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:29,309 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:29,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:29,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095389329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:29,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095389337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:29,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095389437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:29,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095389445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:29,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36666 deadline: 1681095389458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,465 DEBUG [Thread-2306] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:55:29,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:29,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36690 deadline: 1681095389472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:29,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36680 deadline: 1681095389472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,481 DEBUG [Thread-2304] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4177 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:55:29,482 DEBUG [Thread-2302] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:55:29,514 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/82ff01e523fb43eaaa781b75c3890603 2023-04-10 02:55:29,518 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/85eaaf96e11b4253bc9dcea70de49eab as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/85eaaf96e11b4253bc9dcea70de49eab 2023-04-10 02:55:29,530 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/85eaaf96e11b4253bc9dcea70de49eab, entries=150, sequenceid=147, filesize=25.3 K 2023-04-10 02:55:29,531 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/f69e5198878f4887afccc84d08abd2b0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/f69e5198878f4887afccc84d08abd2b0 2023-04-10 02:55:29,538 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/f69e5198878f4887afccc84d08abd2b0, entries=150, sequenceid=147, filesize=11.8 K 2023-04-10 02:55:29,539 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/82ff01e523fb43eaaa781b75c3890603 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/82ff01e523fb43eaaa781b75c3890603 2023-04-10 02:55:29,550 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/82ff01e523fb43eaaa781b75c3890603, entries=150, sequenceid=147, filesize=11.8 K 2023-04-10 02:55:29,551 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for f91f1c15c884b2146b2b8d0ffac5d524 in 897ms, sequenceid=147, compaction requested=true 2023-04-10 02:55:29,551 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:29,551 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:29,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:29,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:29,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:29,555 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:29,555 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:29,555 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:29,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:29,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:29,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@712224f6[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:29,561 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:29,561 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,562 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:29,563 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,563 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:29,564 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:29,565 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:29,565 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:29,565 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:29,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:29,569 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:29,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:29,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:29,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:29,569 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:29,572 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,572 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,572 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:29,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,573 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,575 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,579 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:29,579 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,579 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:29,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:29,579 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:29,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:29,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:29,579 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:29,579 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:29,579 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:29,579 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,579 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,580 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,581 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:29,581 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:29,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:29,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:29,649 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:55:29,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:29,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:29,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:29,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:29,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:29,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:29,681 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:29,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:29,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:29,692 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410a0b842a5b40f4483981e54568d678ab3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a0b842a5b40f4483981e54568d678ab3 2023-04-10 02:55:29,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:29,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:29,696 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:29,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:29,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:29,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:29,697 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,697 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:29,698 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:29,698 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:29,699 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,699 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:29,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:29,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:29,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:29,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:29,728 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:29,728 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:29,728 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:29,728 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:29,728 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:29,728 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:29,728 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:29,728 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:29,728 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:29,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:29,734 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,734 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:29,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095329689, max timestamp=1681095329732Num uniques 859; 2023-04-10 02:55:29,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:29,734 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095329689, max timestamp=1681095329732Num uniques 859; 2023-04-10 02:55:29,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095329689, max timestamp=1681095329732Num uniques 859; 2023-04-10 02:55:29,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:29,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:29,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@354739f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:29,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:29,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,739 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:29,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:29,739 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:29,739 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:29,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:29,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:29,749 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:29,749 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:29,749 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:29,749 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:29,749 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:29,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:29,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:29,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:29,752 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,757 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:29,757 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,757 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:29,757 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:29,757 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:29,757 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:29,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:29,757 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,757 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,757 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,757 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:29,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:29,758 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:29,758 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:29,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:29,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:29,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:29,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095389812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:29,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095389814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,858 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:29,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:29,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:29,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:29,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:29,864 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:29,864 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:29,864 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:29,864 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:29,865 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,865 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:29,866 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:29,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:29,866 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:29,867 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,867 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:29,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:29,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:29,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:29,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:29,868 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:29,868 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:29,868 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:29,872 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:29,872 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:29,872 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:29,872 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:29,874 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:29,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:29,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3422fd2d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:29,877 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:29,877 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,878 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:29,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:29,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:29,878 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,878 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:29,879 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:29,880 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:29,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:29,883 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:29,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:29,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:29,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:29,883 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:29,884 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,884 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:29,884 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:29,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:29,890 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,893 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:29,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:29,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:29,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:29,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:29,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:29,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:29,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:29,898 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:29,899 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,899 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:29,899 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,899 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:29,899 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:29,899 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:29,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:29,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095389921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:29,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095389922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:29,999 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:29,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:30,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:30,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:30,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:30,002 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:30,002 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,002 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:30,002 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:30,003 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,003 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,004 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,004 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,004 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,004 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:30,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:30,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:30,005 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:30,005 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,005 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:30,005 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:30,005 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:30,005 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,005 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:30,006 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:30,006 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:30,007 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,007 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,007 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,007 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:30,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39dfd01c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:30,009 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:30,009 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,010 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:30,010 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,010 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:30,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:30,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:30,012 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:30,016 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:30,016 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:30,016 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:30,016 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:30,016 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:30,018 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,018 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,018 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,018 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,019 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,021 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,021 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,022 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,022 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:30,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:30,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:30,022 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,022 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,022 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,022 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:30,022 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:30,022 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,022 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,022 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,122 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:30,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:30,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:30,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:30,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:30,126 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:30,126 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,126 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:30,126 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:30,126 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,127 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,128 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,128 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,128 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,128 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:30,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:30,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:30,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:30,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:30,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:30,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:30,134 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,134 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:30,134 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:30,134 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,135 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:30,135 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:30,135 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,135 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:30,135 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:30,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,137 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,137 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=158, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/0f18612cceee460c9037656a5b4fceb9 2023-04-10 02:55:30,138 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,138 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:30,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@593a4645[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:30,140 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:30,140 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:30,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,142 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:30,142 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:30,142 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:30,142 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,142 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:30,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095390131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:30,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095390136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,143 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:30,143 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,143 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,143 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:30,146 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:30,146 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:30,146 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:30,146 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:30,146 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:30,147 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,147 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,147 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,148 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,149 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,152 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,153 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:30,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:30,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:30,153 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,153 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,153 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,153 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,153 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,153 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,153 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,153 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,153 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:30,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,153 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:30,253 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:30,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:30,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:30,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:30,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:30,257 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:30,257 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,257 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:30,257 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:30,257 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,257 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,259 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,260 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,260 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:30,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:30,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:30,261 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,261 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:30,261 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:30,261 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:30,264 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,264 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,264 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,265 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:30,266 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79521e76[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:30,266 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,267 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,268 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:30,268 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:30,268 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:30,268 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,268 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:30,270 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,270 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,270 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,270 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:30,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:30,273 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:30,273 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:30,273 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:30,273 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:30,273 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:30,275 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,275 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,281 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,282 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,285 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,289 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,289 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:30,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:30,289 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:30,289 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,289 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,290 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,290 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:30,290 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:30,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,390 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:30,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:30,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:30,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:30,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:30,396 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:30,396 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,396 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:30,396 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:30,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,399 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:30,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:30,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:30,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:30,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:30,405 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:30,405 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:30,405 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,406 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:30,406 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:30,406 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:30,407 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,407 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,408 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,408 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:30,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:30,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a3a4fe4[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:30,409 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:30,409 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,410 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,410 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:30,410 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:30,410 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:30,410 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,410 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:30,412 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:30,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:30,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:30,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:30,414 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:30,414 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:30,414 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:30,414 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:30,414 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:30,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,418 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,419 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,421 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,422 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,422 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,422 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,422 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,422 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,422 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,422 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,422 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,422 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:30,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:30,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,423 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:30,423 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:30,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:30,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095390443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:30,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095390444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,523 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:30,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:30,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:30,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:30,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:30,526 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:30,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:30,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:30,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,527 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,527 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,528 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,528 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:30,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:30,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:30,529 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:30,529 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,529 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:30,529 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:30,529 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:30,529 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,529 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:30,529 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:30,530 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:30,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:30,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:30,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:30,532 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,532 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,532 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,532 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:30,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1da23b8f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:30,534 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:30,534 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,536 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,536 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,536 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:30,536 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:30,536 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:30,536 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:30,538 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,538 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,538 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,538 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:30,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:30,540 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:30,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:30,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:30,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:30,540 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:30,542 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,543 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,543 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,543 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,544 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,547 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:30,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:30,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,547 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:30,547 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:30,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,560 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/1814305a929a4f74bd06685357a4d162 2023-04-10 02:55:30,647 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:30,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:30,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:30,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:30,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:30,651 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:30,651 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,651 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:30,651 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:30,651 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,651 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,653 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,653 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,654 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,654 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:30,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:30,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:30,655 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:30,655 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,655 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:30,656 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:30,656 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:30,656 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,656 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:30,656 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:30,656 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,656 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:30,656 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:30,656 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,656 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:30,657 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:30,658 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,658 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,659 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,659 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:30,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a15e10[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:30,662 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:30,662 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,663 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,663 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:30,663 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:30,663 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,663 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:30,663 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:30,665 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:30,665 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,665 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,665 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:30,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:30,668 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:30,668 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:30,668 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:30,668 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:30,668 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:30,674 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,674 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,674 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,674 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,675 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,678 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:30,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:30,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,678 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:30,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,679 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:30,779 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:30,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:30,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:30,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:30,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:30,782 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:30,782 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,782 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:30,782 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:30,782 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,782 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,786 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,786 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,787 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,787 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:30,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:30,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:30,788 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,788 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:30,788 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:30,788 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,789 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:30,789 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:30,789 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,789 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:30,789 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:30,791 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,792 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:30,793 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:30,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fe9531c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:30,793 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,794 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,794 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:30,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:30,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:30,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:30,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:30,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,795 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,795 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:30,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:30,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:30,798 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:30,798 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:30,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:30,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:30,799 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:30,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,801 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,803 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,807 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,807 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,807 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:30,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:30,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,807 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,808 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:30,808 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,808 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,808 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:30,808 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,808 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,808 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,808 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,908 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:30,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:30,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:30,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:30,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:30,911 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:30,911 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,911 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:30,911 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:30,912 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,912 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,913 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,913 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,913 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,914 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:30,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:30,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:30,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:30,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:30,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:30,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:30,915 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,915 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:30,915 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:30,915 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:30,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:30,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:30,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:30,917 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,917 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,917 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,917 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:30,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a4666a[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:30,919 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:30,919 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,921 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:30,921 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,921 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:30,921 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:30,921 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:30,923 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:30,923 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,923 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,924 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:30,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:30,926 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:30,926 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:30,926 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:30,926 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:30,926 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:30,929 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,929 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:30,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,929 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,930 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:30,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,931 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:30,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:30,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:30,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:30,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:30,933 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:30,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:30,933 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:30,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:30,933 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:30,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:30,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:30,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:30,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095390958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:30,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095390960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:30,970 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/6fda50453ab04466a2f128a0624811ea 2023-04-10 02:55:30,974 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/0f18612cceee460c9037656a5b4fceb9 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/0f18612cceee460c9037656a5b4fceb9 2023-04-10 02:55:30,977 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/0f18612cceee460c9037656a5b4fceb9, entries=200, sequenceid=158, filesize=32.1 K 2023-04-10 02:55:30,978 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/1814305a929a4f74bd06685357a4d162 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/1814305a929a4f74bd06685357a4d162 2023-04-10 02:55:30,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/1814305a929a4f74bd06685357a4d162, entries=150, sequenceid=158, filesize=11.8 K 2023-04-10 02:55:30,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/6fda50453ab04466a2f128a0624811ea as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/6fda50453ab04466a2f128a0624811ea 2023-04-10 02:55:30,985 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/6fda50453ab04466a2f128a0624811ea, entries=150, sequenceid=158, filesize=11.8 K 2023-04-10 02:55:30,986 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for f91f1c15c884b2146b2b8d0ffac5d524 in 1337ms, sequenceid=158, compaction requested=true 2023-04-10 02:55:30,986 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:30,986 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:30,986 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:30,986 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:55:30,986 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:55:30,986 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:30,987 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136123 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:55:30,987 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/A is initiating minor compaction (all files) 2023-04-10 02:55:30,987 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:55:30,987 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/A in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,988 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/B is initiating minor compaction (all files) 2023-04-10 02:55:30,988 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/347c651f9b5544d29c5b2ca52056036d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/543f3193a37d4c64a8dfe76037a3ea52, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/7a8fcf0d20f345dd91a574be55db0ae9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/85eaaf96e11b4253bc9dcea70de49eab, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/0f18612cceee460c9037656a5b4fceb9] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=132.9 K 2023-04-10 02:55:30,988 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/B in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:30,988 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/6ca4706c30244fbe989c279156c73349, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/5066d8615b224398bca9f6f950a696f4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/e87d0eff2f574e9fb313fcbc3578904b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/f69e5198878f4887afccc84d08abd2b0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/1814305a929a4f74bd06685357a4d162] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=58.4 K 2023-04-10 02:55:30,988 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 347c651f9b5544d29c5b2ca52056036d, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681095325212 2023-04-10 02:55:30,988 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6ca4706c30244fbe989c279156c73349, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681095325212 2023-04-10 02:55:30,988 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 543f3193a37d4c64a8dfe76037a3ea52, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681095325296 2023-04-10 02:55:30,988 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 5066d8615b224398bca9f6f950a696f4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681095325296 2023-04-10 02:55:30,989 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 7a8fcf0d20f345dd91a574be55db0ae9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1681095325896 2023-04-10 02:55:30,989 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting e87d0eff2f574e9fb313fcbc3578904b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1681095325896 2023-04-10 02:55:30,989 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 85eaaf96e11b4253bc9dcea70de49eab, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681095327147 2023-04-10 02:55:30,989 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting f69e5198878f4887afccc84d08abd2b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681095327147 2023-04-10 02:55:30,989 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 0f18612cceee460c9037656a5b4fceb9, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681095329317 2023-04-10 02:55:30,989 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 1814305a929a4f74bd06685357a4d162, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681095329317 2023-04-10 02:55:30,998 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#B#compaction#332 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:31,017 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#A#compaction#333 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:31,033 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:31,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:31,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:31,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:31,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:31,037 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:31,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:31,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:31,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:31,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:31,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:31,038 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:31,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:31,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:31,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:31,039 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:31,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:31,039 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:31,039 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:31,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:31,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:31,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:31,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:31,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:31,040 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:31,040 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:31,040 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:31,040 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:31,040 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:31,040 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:31,040 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:31,040 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-10 02:55:31,040 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:31,040 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:31,041 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:31,041 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:31,041 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:31,041 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:31,420 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/1491c7e30dac46e0abb26e5e7b65e7a3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/1491c7e30dac46e0abb26e5e7b65e7a3 2023-04-10 02:55:31,425 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/B of f91f1c15c884b2146b2b8d0ffac5d524 into 1491c7e30dac46e0abb26e5e7b65e7a3(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:31,425 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:31,426 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/B, priority=11, startTime=1681095330986; duration=0sec 2023-04-10 02:55:31,426 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:31,426 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:55:31,427 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:55:31,428 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/C is initiating minor compaction (all files) 2023-04-10 02:55:31,428 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/C in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:31,428 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/3642b6f02e0c40c3b00b9d62fcd4c69e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/48d3a3ada45a4f7295a01323d6fb7067, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe503a2b82d543d2bea92aa077417e85, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/82ff01e523fb43eaaa781b75c3890603, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/6fda50453ab04466a2f128a0624811ea] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=58.4 K 2023-04-10 02:55:31,428 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 3642b6f02e0c40c3b00b9d62fcd4c69e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681095325212 2023-04-10 02:55:31,429 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 48d3a3ada45a4f7295a01323d6fb7067, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681095325296 2023-04-10 02:55:31,429 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting fe503a2b82d543d2bea92aa077417e85, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1681095325896 2023-04-10 02:55:31,429 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 82ff01e523fb43eaaa781b75c3890603, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681095327147 2023-04-10 02:55:31,430 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6fda50453ab04466a2f128a0624811ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681095329317 2023-04-10 02:55:31,445 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#C#compaction#335 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:31,457 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/76ebaf4be0c84d58b8649d8fb3ef2ae4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/76ebaf4be0c84d58b8649d8fb3ef2ae4 2023-04-10 02:55:31,462 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/C of f91f1c15c884b2146b2b8d0ffac5d524 into 76ebaf4be0c84d58b8649d8fb3ef2ae4(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:31,462 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:31,462 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/C, priority=11, startTime=1681095330986; duration=0sec 2023-04-10 02:55:31,462 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:31,465 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410904ac9c13c894b4eac59749377d10f48 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410904ac9c13c894b4eac59749377d10f48 2023-04-10 02:55:31,852 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/e70cfc927c6443d7a1ea38654f9590e1 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/e70cfc927c6443d7a1ea38654f9590e1 2023-04-10 02:55:31,857 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/A of f91f1c15c884b2146b2b8d0ffac5d524 into e70cfc927c6443d7a1ea38654f9590e1(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:31,857 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:31,857 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/A, priority=11, startTime=1681095330986; duration=0sec 2023-04-10 02:55:31,857 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:31,871 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/ddf88e6392fc435d895c1db43b3b87cb 2023-04-10 02:55:31,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:31,966 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:31,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:31,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:31,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095391986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:31,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095391991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:32,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095392097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:32,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095392097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,280 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/ab980306661946e0a5622146e3e421f1 2023-04-10 02:55:32,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:32,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095392300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:32,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095392301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:32,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095392605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:32,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36704 deadline: 1681095392607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,693 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/b2b1cccca4d44883a1ce5c123f79a2bf 2023-04-10 02:55:32,697 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/ddf88e6392fc435d895c1db43b3b87cb as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/ddf88e6392fc435d895c1db43b3b87cb 2023-04-10 02:55:32,700 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/ddf88e6392fc435d895c1db43b3b87cb, entries=150, sequenceid=193, filesize=25.3 K 2023-04-10 02:55:32,701 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/ab980306661946e0a5622146e3e421f1 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/ab980306661946e0a5622146e3e421f1 2023-04-10 02:55:32,707 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/ab980306661946e0a5622146e3e421f1, entries=150, sequenceid=193, filesize=11.8 K 2023-04-10 02:55:32,708 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/b2b1cccca4d44883a1ce5c123f79a2bf as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/b2b1cccca4d44883a1ce5c123f79a2bf 2023-04-10 02:55:32,716 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/b2b1cccca4d44883a1ce5c123f79a2bf, entries=150, sequenceid=193, filesize=11.8 K 2023-04-10 02:55:32,717 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for f91f1c15c884b2146b2b8d0ffac5d524 in 1677ms, sequenceid=193, compaction requested=false 2023-04-10 02:55:32,717 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:32,717 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:32,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:32,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:32,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:32,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:32,720 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,720 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:32,720 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:32,720 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:32,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:32,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bd64552[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:32,722 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:32,722 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:32,724 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:32,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:32,724 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:32,724 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:32,724 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:32,724 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,725 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:32,726 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:32,726 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,726 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:32,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:32,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:32,726 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:32,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:32,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:32,729 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:32,729 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:32,729 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:32,729 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:32,729 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:32,731 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:32,731 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:32,731 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:32,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:32,731 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:32,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,735 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:32,735 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:32,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:32,735 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:32,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:32,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:32,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:32,735 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:32,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:32,736 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:32,736 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:32,735 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:32,735 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:32,736 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,736 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:32,736 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:32,736 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:32,836 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:32,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:32,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:32,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:32,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:32,844 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:32,844 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:32,844 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:32,844 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:32,845 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:32,845 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:32,847 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:32,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:32,848 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:32,848 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:32,848 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:32,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:32,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:32,849 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:32,849 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:32,849 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:32,849 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:32,849 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:32,849 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:32,850 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:32,850 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:55:32,850 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:32,850 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:32,850 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:32,850 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:32,850 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:32,850 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:32,891 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304104303bc5f5f60412d88039ce48bb7a5db to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304104303bc5f5f60412d88039ce48bb7a5db 2023-04-10 02:55:33,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:33,122 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:33,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:33,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36650 deadline: 1681095393238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095333210, max timestamp=1681095333243Num uniques 859; 2023-04-10 02:55:33,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095333210, max timestamp=1681095333243Num uniques 859; 2023-04-10 02:55:33,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:33,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:33,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095333210, max timestamp=1681095333243Num uniques 859; 2023-04-10 02:55:33,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:33,306 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=207, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/00fb516cc3d144f7a9282ca1cf5d2a81 2023-04-10 02:55:33,316 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/28eae182836d400384376f6c8ad0fe2f 2023-04-10 02:55:33,329 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/dd11c398496b422693c8184b07f6387a 2023-04-10 02:55:33,334 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/00fb516cc3d144f7a9282ca1cf5d2a81 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/00fb516cc3d144f7a9282ca1cf5d2a81 2023-04-10 02:55:33,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:33,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095393327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,337 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/00fb516cc3d144f7a9282ca1cf5d2a81, entries=100, sequenceid=207, filesize=18.5 K 2023-04-10 02:55:33,338 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/28eae182836d400384376f6c8ad0fe2f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/28eae182836d400384376f6c8ad0fe2f 2023-04-10 02:55:33,342 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/28eae182836d400384376f6c8ad0fe2f, entries=100, sequenceid=207, filesize=9.4 K 2023-04-10 02:55:33,342 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/dd11c398496b422693c8184b07f6387a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/dd11c398496b422693c8184b07f6387a 2023-04-10 02:55:33,345 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/dd11c398496b422693c8184b07f6387a, entries=100, sequenceid=207, filesize=9.4 K 2023-04-10 02:55:33,346 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for f91f1c15c884b2146b2b8d0ffac5d524 in 496ms, sequenceid=207, compaction requested=true 2023-04-10 02:55:33,346 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:33,346 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:33,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,352 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:33,352 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-10 02:55:33,352 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,352 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:33,352 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:33,352 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:33,352 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:33,352 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:33,352 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:33,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:33,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@679bd5e6[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:33,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:33,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:33,356 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,356 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:33,359 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:33,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:33,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:33,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:33,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:33,362 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:33,363 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:33,363 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:33,363 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:33,363 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:33,365 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,365 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,365 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,366 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,369 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,372 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:33,372 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:33,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:33,372 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,373 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,373 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,373 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,373 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,373 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,373 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:33,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:33,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:33,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:33,373 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:33,373 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:33,373 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:33,379 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304106d10c20f306c44ac8b6ab5998c7ba215 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304106d10c20f306c44ac8b6ab5998c7ba215 2023-04-10 02:55:33,385 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=243, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/b1d44c6e244e4818bd398bb4c409c9c0 2023-04-10 02:55:33,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:33,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095393407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:33,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095393435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,473 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:33,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:33,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:33,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:33,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:33,478 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:33,478 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:33,478 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:33,478 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:33,478 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,478 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,480 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,480 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:33,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:33,481 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,481 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:33,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:33,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:33,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:33,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:33,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:33,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:33,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:33,484 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:33,484 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:33,484 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:33,484 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:33,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:33,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:33,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:33,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:33,486 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,486 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,487 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,487 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:33,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@525f0bba[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:33,488 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:33,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,491 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:33,491 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:33,491 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:33,491 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:33,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:33,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36666 deadline: 1681095393488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,493 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,494 DEBUG [Thread-2306] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8184 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:55:33,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:33,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:33,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36680 deadline: 1681095393491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:33,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:33,502 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:33,502 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:33,502 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:33,502 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:33,502 DEBUG [Thread-2302] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8190 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:55:33,502 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:33,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:33,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36690 deadline: 1681095393499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:33,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,506 DEBUG [Thread-2304] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8201 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:55:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:33,506 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,508 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,511 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:33,511 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:33,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:33,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:33,511 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,511 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,511 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,511 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,511 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:33,511 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,511 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,511 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:33,511 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:33,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:33,511 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:33,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:33,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:33,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095393516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,612 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:33,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:33,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:33,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:33,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:33,615 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:33,615 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:33,615 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:33,615 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:33,615 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,615 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,617 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:33,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:33,617 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,618 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,618 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:33,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:33,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:33,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:33,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:33,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:33,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:33,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:33,620 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:33,620 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:33,620 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:33,620 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:33,620 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:33,620 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:33,620 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,620 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:33,620 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:33,621 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,622 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,622 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,622 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:33,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61ca0ad[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:33,623 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:33,623 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,625 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,625 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:33,625 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:33,625 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:33,626 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,626 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:33,628 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:33,628 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:33,628 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,628 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:33,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:33,631 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:33,631 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:33,631 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:33,631 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:33,631 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:33,641 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,642 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,642 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,642 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:33,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:33,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:33,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,644 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:33,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095393642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,646 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:33,646 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:33,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:33,646 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:33,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:33,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:33,647 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:33,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:33,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,647 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:33,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,647 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:33,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:33,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:33,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095393735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,747 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:33,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:33,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:33,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:33,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:33,750 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:33,750 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:33,750 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:33,750 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:33,751 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,751 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,752 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,752 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:33,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:33,752 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,752 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:33,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:33,753 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:33,753 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:33,753 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:33,753 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:33,753 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:33,754 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:33,754 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:33,754 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:33,754 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:33,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:33,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:33,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:33,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:33,756 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,756 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,756 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:33,758 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:33,758 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fa999e9[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:33,759 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,759 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:33,759 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:33,759 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,759 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:33,759 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:33,763 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:33,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:33,765 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:33,765 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:33,765 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:33,765 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:33,765 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:33,772 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,772 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,773 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,773 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,774 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,777 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,777 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,777 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:33,777 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:33,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:33,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:33,777 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:33,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:33,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:33,777 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,778 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:33,778 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,778 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:33,778 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:33,778 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,778 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,796 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/3d6d5c711cfe47cf83132cd75b4788cf 2023-04-10 02:55:33,878 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:33,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:33,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:33,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:33,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:33,881 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:33,881 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:33,881 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:33,881 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:33,882 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,882 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,883 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:33,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:33,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:33,884 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,884 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:33,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:33,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:33,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:33,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:33,886 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:33,886 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:33,886 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:33,886 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:33,886 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:33,886 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:33,887 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:33,887 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:33,887 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:33,887 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:33,887 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:33,887 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,887 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:33,887 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:33,888 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,888 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,889 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,889 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:33,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76a4511f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:33,890 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:33,890 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,892 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,892 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:33,892 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,892 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:33,892 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:33,892 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:33,894 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:33,894 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,894 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,894 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:33,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:33,896 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:33,896 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:33,896 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:33,896 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:33,896 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:33,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:33,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:33,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:33,899 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,900 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,902 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,902 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:33,902 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:33,902 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:33,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:33,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:33,902 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:33,902 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:33,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:33,902 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:33,902 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:33,902 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:33,902 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,903 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:33,903 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:33,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:33,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095393948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,002 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:34,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:34,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:34,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:34,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:34,006 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:34,006 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:34,006 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:34,006 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:34,006 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,006 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:34,008 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,008 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:34,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:34,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:34,008 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,008 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:34,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:34,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:34,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:34,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:34,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:34,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:34,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:34,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:34,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:34,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:34,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:34,010 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:34,010 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:34,010 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:34,010 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:34,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:34,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:34,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:34,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:34,012 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,012 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:34,012 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:34,012 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:34,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:34,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:34,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:34,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:34,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0c96aa[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:34,017 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:34,017 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,018 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,018 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:34,019 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:34,019 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:34,019 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,019 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:34,020 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:34,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:34,020 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:34,020 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:34,020 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:34,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:34,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:34,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:34,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:34,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:34,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:34,023 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:34,023 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:34,023 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:34,023 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:34,023 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:34,025 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,025 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:34,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:34,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,025 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:34,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:34,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:34,025 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,027 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:34,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:34,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:34,030 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:34,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:34,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:34,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:34,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:34,031 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:34,031 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:34,031 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:34,031 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:34,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:34,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:34,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095394041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,131 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:34,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:34,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:34,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:34,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:34,134 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:34,134 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:34,134 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:34,134 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:34,135 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,135 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:34,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:34,137 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:34,138 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,138 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:34,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:34,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:34,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:34,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:34,138 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:34,138 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:34,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:34,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:34,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:34,139 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:34,139 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:34,139 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:34,139 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:34,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:34,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:34,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:34,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:34,142 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,142 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:34,143 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:34,143 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:34,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:34,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f764c4e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:34,144 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:34,144 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,147 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,147 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:34,147 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:34,147 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:34,147 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,147 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:34,149 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:34,149 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:34,149 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,149 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:34,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:34,151 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:34,151 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:34,151 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:34,151 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:34,151 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:34,156 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,156 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:34,156 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,156 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,157 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:34,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:34,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,165 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:34,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:34,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:34,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:34,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:34,166 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:34,166 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:34,166 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:34,166 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:34,166 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:34,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:34,207 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/685b63b5f0ea4546867ee37fd7789a41 2023-04-10 02:55:34,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/b1d44c6e244e4818bd398bb4c409c9c0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/b1d44c6e244e4818bd398bb4c409c9c0 2023-04-10 02:55:34,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/b1d44c6e244e4818bd398bb4c409c9c0, entries=200, sequenceid=243, filesize=32.1 K 2023-04-10 02:55:34,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/3d6d5c711cfe47cf83132cd75b4788cf as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/3d6d5c711cfe47cf83132cd75b4788cf 2023-04-10 02:55:34,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/3d6d5c711cfe47cf83132cd75b4788cf, entries=150, sequenceid=243, filesize=11.8 K 2023-04-10 02:55:34,219 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/685b63b5f0ea4546867ee37fd7789a41 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/685b63b5f0ea4546867ee37fd7789a41 2023-04-10 02:55:34,226 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/685b63b5f0ea4546867ee37fd7789a41, entries=150, sequenceid=243, filesize=11.8 K 2023-04-10 02:55:34,226 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for f91f1c15c884b2146b2b8d0ffac5d524 in 874ms, sequenceid=243, compaction requested=true 2023-04-10 02:55:34,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:34,227 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:55:34,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:34,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:34,227 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:55:34,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:34,228 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103850 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:55:34,228 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/A is initiating minor compaction (all files) 2023-04-10 02:55:34,228 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/A in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:34,228 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/e70cfc927c6443d7a1ea38654f9590e1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/ddf88e6392fc435d895c1db43b3b87cb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/00fb516cc3d144f7a9282ca1cf5d2a81, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/b1d44c6e244e4818bd398bb4c409c9c0] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=101.4 K 2023-04-10 02:55:34,229 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting e70cfc927c6443d7a1ea38654f9590e1, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681095329317 2023-04-10 02:55:34,229 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46024 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:55:34,229 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/B is initiating minor compaction (all files) 2023-04-10 02:55:34,229 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/B in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:34,229 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/1491c7e30dac46e0abb26e5e7b65e7a3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/ab980306661946e0a5622146e3e421f1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/28eae182836d400384376f6c8ad0fe2f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/3d6d5c711cfe47cf83132cd75b4788cf] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=44.9 K 2023-04-10 02:55:34,229 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ddf88e6392fc435d895c1db43b3b87cb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681095329789 2023-04-10 02:55:34,230 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 1491c7e30dac46e0abb26e5e7b65e7a3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681095329317 2023-04-10 02:55:34,230 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 00fb516cc3d144f7a9282ca1cf5d2a81, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681095331985 2023-04-10 02:55:34,230 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting ab980306661946e0a5622146e3e421f1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681095329789 2023-04-10 02:55:34,230 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting b1d44c6e244e4818bd398bb4c409c9c0, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681095333259 2023-04-10 02:55:34,230 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 28eae182836d400384376f6c8ad0fe2f, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681095331985 2023-04-10 02:55:34,231 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 3d6d5c711cfe47cf83132cd75b4788cf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681095333308 2023-04-10 02:55:34,238 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#B#compaction#345 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:34,244 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#A#compaction#344 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:34,252 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/08060eb67abd49e08cb150146dacdb48 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/08060eb67abd49e08cb150146dacdb48 2023-04-10 02:55:34,255 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/B of f91f1c15c884b2146b2b8d0ffac5d524 into 08060eb67abd49e08cb150146dacdb48(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:34,256 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:34,256 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/B, priority=12, startTime=1681095334227; duration=0sec 2023-04-10 02:55:34,256 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:34,256 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:55:34,257 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46024 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:55:34,257 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/C is initiating minor compaction (all files) 2023-04-10 02:55:34,257 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/C in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:34,257 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/76ebaf4be0c84d58b8649d8fb3ef2ae4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/b2b1cccca4d44883a1ce5c123f79a2bf, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/dd11c398496b422693c8184b07f6387a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/685b63b5f0ea4546867ee37fd7789a41] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=44.9 K 2023-04-10 02:55:34,258 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 76ebaf4be0c84d58b8649d8fb3ef2ae4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681095329317 2023-04-10 02:55:34,258 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting b2b1cccca4d44883a1ce5c123f79a2bf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681095329789 2023-04-10 02:55:34,258 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting dd11c398496b422693c8184b07f6387a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681095331985 2023-04-10 02:55:34,258 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 685b63b5f0ea4546867ee37fd7789a41, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681095333308 2023-04-10 02:55:34,266 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#C#compaction#346 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:34,266 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:34,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:34,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:34,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:34,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:34,270 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:34,270 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:34,270 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:34,270 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:34,271 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,271 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:34,273 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:34,273 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:34,273 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/fe1d07ca99434fda9ef25a3b4614b48d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe1d07ca99434fda9ef25a3b4614b48d 2023-04-10 02:55:34,274 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,274 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:34,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:34,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:34,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:34,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:34,275 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:34,275 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:34,275 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:34,275 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:34,275 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:34,275 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:34,275 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:34,275 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:55:34,275 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:34,276 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:34,276 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:34,276 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:34,276 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:34,276 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:34,277 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/C of f91f1c15c884b2146b2b8d0ffac5d524 into fe1d07ca99434fda9ef25a3b4614b48d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:34,277 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:34,277 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/C, priority=12, startTime=1681095334227; duration=0sec 2023-04-10 02:55:34,277 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:34,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:34,464 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:34,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095334538, max timestamp=1681095334555Num uniques 859; 2023-04-10 02:55:34,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:34,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095334538, max timestamp=1681095334555Num uniques 859; 2023-04-10 02:55:34,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095334538, max timestamp=1681095334555Num uniques 859; 2023-04-10 02:55:34,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:34,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:34,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:34,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095394602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:34,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095394603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,688 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304109c39d6fdc777433d917015ee8d1841eb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304109c39d6fdc777433d917015ee8d1841eb 2023-04-10 02:55:34,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:34,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095394710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:34,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095394710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,716 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=254, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/c180bd081ed34487a3e81343e7e38282 2023-04-10 02:55:34,735 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/07400ba0fd4044c1bf44708950313a53 2023-04-10 02:55:34,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:34,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095394914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:34,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:34,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095394914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,078 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/7e3de2cd68da40b2b85629eb314e7137 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/7e3de2cd68da40b2b85629eb314e7137 2023-04-10 02:55:35,082 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/A of f91f1c15c884b2146b2b8d0ffac5d524 into 7e3de2cd68da40b2b85629eb314e7137(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:35,082 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:35,082 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/A, priority=12, startTime=1681095334226; duration=0sec 2023-04-10 02:55:35,083 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:35,154 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/fe8e35916aef4ee9800d4b104387ace6 2023-04-10 02:55:35,158 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/c180bd081ed34487a3e81343e7e38282 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/c180bd081ed34487a3e81343e7e38282 2023-04-10 02:55:35,161 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/c180bd081ed34487a3e81343e7e38282, entries=100, sequenceid=254, filesize=18.5 K 2023-04-10 02:55:35,162 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/07400ba0fd4044c1bf44708950313a53 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/07400ba0fd4044c1bf44708950313a53 2023-04-10 02:55:35,165 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/07400ba0fd4044c1bf44708950313a53, entries=100, sequenceid=254, filesize=9.4 K 2023-04-10 02:55:35,166 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/fe8e35916aef4ee9800d4b104387ace6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe8e35916aef4ee9800d4b104387ace6 2023-04-10 02:55:35,169 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe8e35916aef4ee9800d4b104387ace6, entries=100, sequenceid=254, filesize=9.4 K 2023-04-10 02:55:35,169 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f91f1c15c884b2146b2b8d0ffac5d524 in 894ms, sequenceid=254, compaction requested=false 2023-04-10 02:55:35,169 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:35,169 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:35,169 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:35,169 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:35,169 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,170 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:35,170 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:35,171 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,171 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,172 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,172 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:35,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c93fc98[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:35,173 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:35,173 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,174 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,174 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:35,174 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:35,174 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:35,174 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,174 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:35,178 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:35,178 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,178 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,178 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:35,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:35,180 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:35,180 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:35,180 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:35,180 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:35,180 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:35,182 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,182 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,182 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:35,182 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,183 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,185 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,185 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,185 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,185 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,185 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,185 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:35,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:35,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:35,185 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,186 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,185 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:35,186 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:35,186 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:35,186 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:35,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:35,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:35,222 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-10 02:55:35,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:35,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:35,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:35,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:35,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:35,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:35,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:35,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095395236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:35,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095395238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,286 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:35,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:35,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:35,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:35,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:35,290 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:35,290 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:35,290 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:35,290 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:35,290 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,290 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,291 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,291 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:35,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:35,292 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,292 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:35,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:35,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:35,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:35,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:35,293 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:35,293 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:35,293 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:35,293 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:35,293 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:35,293 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:35,293 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:35,293 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:35,293 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:35,293 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:35,293 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:35,293 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,294 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:35,294 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:35,296 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,296 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,296 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,296 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:35,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:35,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@686b6ebc[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:35,298 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:35,298 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,299 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:35,299 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:35,299 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:35,299 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,299 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:35,301 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,301 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:35,301 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,301 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:35,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:35,303 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:35,303 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:35,303 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:35,303 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:35,303 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:35,305 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,305 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,305 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:35,305 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,306 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,309 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,309 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:35,309 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,309 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,309 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:35,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:35,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:35,309 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,309 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:35,309 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:35,309 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,309 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:35,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:35,309 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,310 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:35,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095395338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:35,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095395343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,410 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:35,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:35,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:35,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:35,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:35,413 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:35,413 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:35,413 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:35,413 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:35,413 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,413 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,414 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:35,414 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:35,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:35,415 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,415 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:35,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:35,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:35,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:35,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:35,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:35,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:35,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:35,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:35,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:35,416 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:35,416 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:35,416 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:35,416 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:35,417 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:35,417 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:35,417 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,417 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:35,417 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:35,418 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,418 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,418 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,418 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:35,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:35,420 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:35,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50ea1cdd[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:35,420 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,421 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,421 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:35,421 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:35,421 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:35,421 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,421 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:35,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:35,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:35,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:35,426 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:35,426 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:35,426 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:35,426 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:35,426 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:35,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:35,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,429 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,430 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:35,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:35,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:35,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:35,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:35,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,433 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:35,433 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:35,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,433 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:35,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,434 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,534 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:35,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:35,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:35,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:35,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:35,538 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:35,538 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:35,538 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:35,538 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:35,538 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,538 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,541 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:35,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:35,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:35,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:35,544 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:35,544 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:35,544 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:35,545 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:35,548 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:35,548 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:35,548 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:35,548 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:35,550 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,550 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:35,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:35,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095395543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:35,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d733eb2[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:35,553 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:35,553 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,554 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,554 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:35,554 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:35,554 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:35,555 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,555 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:35,556 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:35,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:35,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095395550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:35,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:35,560 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:35,560 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:35,560 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:35,560 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:35,560 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:35,563 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,563 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,563 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:35,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,564 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,565 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,567 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,567 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,567 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:35,567 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,567 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:35,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:35,567 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:35,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:35,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,567 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:35,568 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,568 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:35,568 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,568 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,568 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:35,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:35,635 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410585619c2b93f47a8ad1302e6de5ad203 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410585619c2b93f47a8ad1302e6de5ad203 2023-04-10 02:55:35,641 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=293, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/5e512f0191704438bf32156f58ea4a7d 2023-04-10 02:55:35,668 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:35,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:35,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:35,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:35,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:35,671 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:35,671 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:35,671 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:35,671 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:35,671 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,671 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,673 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:35,673 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:35,673 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,673 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:35,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:35,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:35,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:35,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:35,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:35,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:35,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:35,674 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:35,674 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:35,675 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:35,675 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:35,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:35,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:35,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:35,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:35,676 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:35,678 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:35,678 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e276eb8[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:35,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:35,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:35,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:35,680 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,680 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:35,681 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:35,683 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:35,683 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:35,683 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:35,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:35,684 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:35,685 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,685 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,685 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:35,686 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,687 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,688 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,689 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,689 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,689 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:35,689 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,689 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,689 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,689 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:35,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:35,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:35,689 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,689 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,689 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:35,689 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:35,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:35,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:35,789 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:35,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:35,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:35,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:35,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:35,793 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:35,793 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:35,793 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:35,793 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:35,793 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,793 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,796 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:35,796 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:35,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:35,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:35,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:35,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:35,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:35,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:35,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:35,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:35,798 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:35,798 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:35,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:35,798 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:35,798 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:35,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:35,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:35,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:35,798 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:35,800 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,800 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,800 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,800 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:35,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0ae14e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:35,801 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:35,802 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,802 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,803 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:35,803 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:35,803 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:35,803 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,803 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:35,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:35,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:35,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:35,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:35,806 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:35,806 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:35,806 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:35,807 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:35,807 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:35,809 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,809 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,809 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,811 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,813 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:35,813 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,813 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:35,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:35,813 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:35,813 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:35,813 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:35,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:35,813 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,814 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:35,813 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:35,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095395853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:35,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095395857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,914 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:35,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:35,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:35,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:35,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:35,917 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:35,917 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:35,917 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:35,917 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:35,918 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,918 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,923 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:35,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:35,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:35,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:35,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:35,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:35,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:35,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:35,924 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:35,924 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:35,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:35,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:35,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:35,925 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:35,925 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:35,925 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:35,925 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:35,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:35,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:35,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:35,925 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:35,927 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,927 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,927 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,927 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:35,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0600c6[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:35,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:35,929 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,930 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,930 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:35,930 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:35,930 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:35,930 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,930 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:35,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:35,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:35,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:35,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:35,936 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:35,936 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:35,936 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:35,936 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:35,936 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:35,937 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,937 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,938 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:35,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:35,938 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,940 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:35,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:35,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:35,942 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:35,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:35,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:35,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:35,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:35,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:35,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:35,943 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:35,943 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,943 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:35,943 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:35,943 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:35,943 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:35,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:35,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:36,043 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:36,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:36,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:36,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:36,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:36,046 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:36,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:36,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:36,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:36,047 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:36,047 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:36,048 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:36,048 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:36,048 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,049 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:36,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:36,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:36,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:36,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:36,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:36,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:36,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:36,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:36,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:36,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:36,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:36,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:36,050 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:36,050 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:36,050 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:36,050 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:36,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:36,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:36,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:36,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:36,051 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/e3c423518f7d4916aa68303fc9da2a49 2023-04-10 02:55:36,057 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,057 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:36,057 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:36,057 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:36,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:36,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9c4317a[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:36,059 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:36,059 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:36,060 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:36,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:36,060 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:36,060 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:36,060 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:36,061 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,061 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:36,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/41a3c24c18514ca188106950efd5166f 2023-04-10 02:55:36,062 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,062 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:36,062 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:36,062 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:36,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:36,065 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:36,065 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:36,065 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:36,065 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:36,065 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:36,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/5e512f0191704438bf32156f58ea4a7d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/5e512f0191704438bf32156f58ea4a7d 2023-04-10 02:55:36,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:36,067 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:36,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:36,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:36,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:36,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:36,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:36,068 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/5e512f0191704438bf32156f58ea4a7d, entries=200, sequenceid=293, filesize=32.3 K 2023-04-10 02:55:36,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/e3c423518f7d4916aa68303fc9da2a49 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/e3c423518f7d4916aa68303fc9da2a49 2023-04-10 02:55:36,070 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:36,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:36,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:36,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:36,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:36,071 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:36,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:36,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:36,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,073 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,073 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/e3c423518f7d4916aa68303fc9da2a49, entries=150, sequenceid=293, filesize=11.9 K 2023-04-10 02:55:36,073 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/41a3c24c18514ca188106950efd5166f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/41a3c24c18514ca188106950efd5166f 2023-04-10 02:55:36,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:36,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:36,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/41a3c24c18514ca188106950efd5166f, entries=150, sequenceid=293, filesize=11.9 K 2023-04-10 02:55:36,082 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:36,082 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:36,082 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:36,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:36,082 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:36,082 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for f91f1c15c884b2146b2b8d0ffac5d524 in 860ms, sequenceid=293, compaction requested=true 2023-04-10 02:55:36,082 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:36,082 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:36,082 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,082 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:36,082 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:36,082 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:36,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:36,083 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:36,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:36,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:36,083 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:36,083 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-10 02:55:36,083 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:36,083 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:36,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:36,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:36,084 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78290 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:36,084 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/A is initiating minor compaction (all files) 2023-04-10 02:55:36,084 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:36,084 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/A in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:36,084 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:36,084 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/7e3de2cd68da40b2b85629eb314e7137, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/c180bd081ed34487a3e81343e7e38282, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/5e512f0191704438bf32156f58ea4a7d] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=76.5 K 2023-04-10 02:55:36,085 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:36,085 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/B is initiating minor compaction (all files) 2023-04-10 02:55:36,085 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/B in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:36,086 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/08060eb67abd49e08cb150146dacdb48, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/07400ba0fd4044c1bf44708950313a53, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/e3c423518f7d4916aa68303fc9da2a49] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=33.5 K 2023-04-10 02:55:36,086 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 7e3de2cd68da40b2b85629eb314e7137, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681095333308 2023-04-10 02:55:36,086 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 08060eb67abd49e08cb150146dacdb48, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681095333308 2023-04-10 02:55:36,086 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting c180bd081ed34487a3e81343e7e38282, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681095333401 2023-04-10 02:55:36,086 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 07400ba0fd4044c1bf44708950313a53, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681095333401 2023-04-10 02:55:36,087 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 5e512f0191704438bf32156f58ea4a7d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681095334584 2023-04-10 02:55:36,087 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting e3c423518f7d4916aa68303fc9da2a49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681095334601 2023-04-10 02:55:36,094 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#B#compaction#354 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:36,098 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#A#compaction#353 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:36,184 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:36,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:36,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:36,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:36,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:36,187 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:36,187 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:36,187 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:36,187 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:36,188 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:36,188 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:36,189 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,189 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:36,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:36,189 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,189 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:36,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:36,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:36,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:36,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:36,190 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:36,190 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:36,190 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:36,190 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:36,190 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:36,191 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:36,191 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:36,191 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-10 02:55:36,191 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:36,191 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:36,191 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:36,191 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:36,191 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:36,191 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:36,207 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041088ec34eee4064610948089205c231d2e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041088ec34eee4064610948089205c231d2e 2023-04-10 02:55:36,212 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=302, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/33a4a252f7c84aeba58a36785625ef3e 2023-04-10 02:55:36,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:36,365 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:36,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095336397, max timestamp=1681095336416Num uniques 859; 2023-04-10 02:55:36,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095336397, max timestamp=1681095336416Num uniques 859; 2023-04-10 02:55:36,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:36,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095336397, max timestamp=1681095336416Num uniques 859; 2023-04-10 02:55:36,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:36,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:36,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:36,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095396484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:36,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095396492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,527 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/abba928c353f49a58dbcba78dd3c0b0c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/abba928c353f49a58dbcba78dd3c0b0c 2023-04-10 02:55:36,531 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/B of f91f1c15c884b2146b2b8d0ffac5d524 into abba928c353f49a58dbcba78dd3c0b0c(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:36,531 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:36,531 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/B, priority=13, startTime=1681095336083; duration=0sec 2023-04-10 02:55:36,531 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:36,531 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:36,532 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:36,532 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/C is initiating minor compaction (all files) 2023-04-10 02:55:36,532 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/C in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:36,532 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe1d07ca99434fda9ef25a3b4614b48d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe8e35916aef4ee9800d4b104387ace6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/41a3c24c18514ca188106950efd5166f] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=33.5 K 2023-04-10 02:55:36,533 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting fe1d07ca99434fda9ef25a3b4614b48d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681095333308 2023-04-10 02:55:36,533 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting fe8e35916aef4ee9800d4b104387ace6, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681095333401 2023-04-10 02:55:36,533 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 41a3c24c18514ca188106950efd5166f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681095334601 2023-04-10 02:55:36,539 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#C#compaction#357 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:36,551 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/3ae4cf2f037d4f38a08bd15e0c2c59cd as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/3ae4cf2f037d4f38a08bd15e0c2c59cd 2023-04-10 02:55:36,554 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/C of f91f1c15c884b2146b2b8d0ffac5d524 into 3ae4cf2f037d4f38a08bd15e0c2c59cd(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:36,555 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:36,555 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/C, priority=13, startTime=1681095336083; duration=0sec 2023-04-10 02:55:36,555 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:36,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:36,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095396593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:36,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095396603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,622 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/c209ee994b5448f5875b349f7f268b0c 2023-04-10 02:55:36,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:36,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095396800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:36,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:36,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095396814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,030 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/59a0606c07684d8c9ce73d08ce673906 2023-04-10 02:55:37,034 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/33a4a252f7c84aeba58a36785625ef3e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/33a4a252f7c84aeba58a36785625ef3e 2023-04-10 02:55:37,038 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/33a4a252f7c84aeba58a36785625ef3e, entries=150, sequenceid=302, filesize=25.4 K 2023-04-10 02:55:37,039 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/c209ee994b5448f5875b349f7f268b0c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/c209ee994b5448f5875b349f7f268b0c 2023-04-10 02:55:37,043 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/c209ee994b5448f5875b349f7f268b0c, entries=150, sequenceid=302, filesize=11.9 K 2023-04-10 02:55:37,043 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/59a0606c07684d8c9ce73d08ce673906 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/59a0606c07684d8c9ce73d08ce673906 2023-04-10 02:55:37,047 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/59a0606c07684d8c9ce73d08ce673906, entries=150, sequenceid=302, filesize=11.9 K 2023-04-10 02:55:37,048 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for f91f1c15c884b2146b2b8d0ffac5d524 in 857ms, sequenceid=302, compaction requested=false 2023-04-10 02:55:37,048 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:37,048 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:37,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:37,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:37,048 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:37,049 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,049 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:37,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c53dc54[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:37,053 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:37,053 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,054 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,054 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:37,054 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:37,054 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:37,055 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,055 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:37,057 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,057 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:37,057 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,057 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:37,059 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:37,059 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:37,059 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:37,059 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:37,059 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:37,061 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,061 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,061 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,062 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,063 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,065 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,065 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,065 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,065 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:37,065 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,065 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:37,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:37,065 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,066 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,066 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,066 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,066 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:37,066 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:37,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:37,107 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-10 02:55:37,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:37,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:55:37,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:37,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:55:37,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:37,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:55:37,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:37,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095397129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:37,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095397130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,166 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:37,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:37,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:37,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:37,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:37,170 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:37,170 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,170 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:37,170 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:37,170 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,170 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,171 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,171 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,172 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,172 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:37,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:37,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:37,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:37,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:37,176 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:37,176 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,176 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:37,176 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:37,176 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:37,176 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,176 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:37,176 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:37,177 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,177 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:37,177 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:37,177 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,177 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:37,177 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:37,184 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,185 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,185 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,185 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:37,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56c94401[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:37,188 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:37,188 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,189 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:37,190 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,190 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:37,192 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:37,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:37,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:37,196 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:37,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:37,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:37,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:37,196 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:37,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,199 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,200 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:37,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:37,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:37,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,202 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,203 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,203 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,203 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:37,203 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,203 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:37,203 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:37,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095397235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:37,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095397239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,303 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:37,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:37,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:37,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:37,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:37,306 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:37,306 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,306 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:37,306 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:37,307 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,307 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,308 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,308 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,309 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,309 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:37,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:37,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:37,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:37,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:37,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:37,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:37,310 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,310 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:37,310 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:37,310 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:37,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:37,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:37,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:37,312 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,312 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,312 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,312 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:37,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:37,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b346e2e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:37,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,315 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:37,315 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,315 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:37,316 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:37,316 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,316 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,316 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:37,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:37,318 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:37,319 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:37,319 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:37,319 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:37,319 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:37,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,321 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,323 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,324 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,324 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,324 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:37,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:37,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,325 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:37,325 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:37,324 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,324 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,325 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,325 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,325 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,325 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,325 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,337 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/521fae27b38541ec997ce5b807db4120 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/521fae27b38541ec997ce5b807db4120 2023-04-10 02:55:37,341 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/A of f91f1c15c884b2146b2b8d0ffac5d524 into 521fae27b38541ec997ce5b807db4120(size=25.9 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-10 02:55:37,341 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:37,341 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/A, priority=13, startTime=1681095336082; duration=1sec 2023-04-10 02:55:37,341 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:37,425 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:37,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:37,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:37,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:37,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:37,428 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:37,428 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,429 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:37,429 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:37,429 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,429 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,430 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,431 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,431 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:37,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:37,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:37,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:37,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:37,432 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:37,432 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,432 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:37,432 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:37,432 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:37,432 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,432 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:37,432 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:37,432 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,433 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:37,433 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:37,433 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,433 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:37,433 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:37,434 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,434 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,434 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,434 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:37,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d3c5d3[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:37,436 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:37,436 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,437 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:37,438 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,438 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:37,439 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:37,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:37,442 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:37,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:37,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:37,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:37,442 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:37,443 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,443 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,443 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,444 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,445 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:37,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095397443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,447 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,447 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,447 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,447 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,447 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:37,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,447 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,447 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:37,448 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,447 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,448 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:37,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,448 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:37,448 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:37,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:37,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095397449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,522 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410a070f30322d84ce3a1e299f8c9e12790 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a070f30322d84ce3a1e299f8c9e12790 2023-04-10 02:55:37,548 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:37,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:37,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:37,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:37,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:37,551 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:37,551 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,551 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:37,551 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:37,552 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,552 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,554 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,554 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,554 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,554 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:37,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:37,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:37,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:37,555 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:37,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:37,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:37,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:37,556 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,556 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:37,556 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:37,556 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:37,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:37,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:37,556 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:37,558 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,558 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,558 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,558 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:37,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:37,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51b01881[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:37,559 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:37,559 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,560 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:37,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:37,561 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:37,561 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,561 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:37,565 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,565 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:37,565 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,565 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:37,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:37,568 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:37,568 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:37,568 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:37,568 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:37,568 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:37,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,570 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,570 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,571 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,573 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,573 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,573 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:37,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:37,573 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,573 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,573 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,574 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:37,574 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:37,574 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,574 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,574 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,574 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,674 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:37,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:37,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:37,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:37,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:37,677 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:37,677 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,677 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:37,677 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:37,677 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,677 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,678 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,679 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,679 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:37,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:37,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:37,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:37,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:37,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:37,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:37,680 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,680 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:37,680 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:37,680 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:37,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:37,680 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:37,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:37,683 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,683 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,683 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,683 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:37,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dc5b7fa[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:37,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:37,685 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,686 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,686 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:37,686 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,686 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:37,686 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:37,686 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:37,688 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,688 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:37,688 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,688 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:37,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:37,690 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:37,690 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:37,690 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:37,690 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:37,690 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:37,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,692 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,693 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:37,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:37,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:37,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,697 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:37,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,697 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:37,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:37,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095397748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:37,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095397754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,797 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:37,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:37,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:37,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:37,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:37,800 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:37,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:37,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:37,801 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,801 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,802 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,802 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,802 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,802 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:37,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:37,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:37,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:37,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:37,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:37,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:37,805 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:37,805 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:37,805 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,805 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:37,805 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:37,805 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,805 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:37,805 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:37,805 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,805 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:37,805 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:37,807 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:37,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4508f572[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:37,809 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:37,809 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,810 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:37,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:37,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:37,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:37,813 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:37,813 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,813 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,813 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:37,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:37,816 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:37,816 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:37,816 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:37,816 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:37,816 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:37,817 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,817 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,817 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,818 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,821 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,821 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,821 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:37,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:37,821 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:37,822 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:37,822 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:37,821 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,821 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,822 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,822 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,822 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,822 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,922 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:37,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:37,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:37,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:37,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:37,925 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:37,925 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,925 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:37,925 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:37,926 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,926 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,927 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,927 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,927 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,927 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:37,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,928 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=342, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/9671db15da2341dc83a2dc53e4abeb26 2023-04-10 02:55:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:37,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:37,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:37,928 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:37,928 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:37,928 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:37,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:37,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:37,929 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,929 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:37,929 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:37,929 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:37,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:37,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:37,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:37,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,931 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:37,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e85d5d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:37,935 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:37,935 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,936 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:37,936 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:37,936 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,936 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:37,936 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:37,938 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:37,938 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,938 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:37,938 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,939 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/5f3ef065621548d797da3969ab747299 2023-04-10 02:55:37,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:37,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:37,944 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:37,944 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:37,944 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:37,944 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:37,944 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:37,949 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,949 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,949 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:37,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:37,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,952 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:37,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:37,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:37,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:37,956 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:37,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:37,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:37,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:37,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,956 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:37,956 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:37,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:37,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:37,957 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:37,957 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,957 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:37,957 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:37,961 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/cd6a8d0a721a4b97af7bc1d3d33d7045 2023-04-10 02:55:37,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/9671db15da2341dc83a2dc53e4abeb26 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/9671db15da2341dc83a2dc53e4abeb26 2023-04-10 02:55:37,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/9671db15da2341dc83a2dc53e4abeb26, entries=200, sequenceid=342, filesize=32.3 K 2023-04-10 02:55:37,970 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/5f3ef065621548d797da3969ab747299 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/5f3ef065621548d797da3969ab747299 2023-04-10 02:55:37,974 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/5f3ef065621548d797da3969ab747299, entries=150, sequenceid=342, filesize=11.9 K 2023-04-10 02:55:37,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/cd6a8d0a721a4b97af7bc1d3d33d7045 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/cd6a8d0a721a4b97af7bc1d3d33d7045 2023-04-10 02:55:37,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/cd6a8d0a721a4b97af7bc1d3d33d7045, entries=150, sequenceid=342, filesize=11.9 K 2023-04-10 02:55:37,979 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for f91f1c15c884b2146b2b8d0ffac5d524 in 871ms, sequenceid=342, compaction requested=true 2023-04-10 02:55:37,979 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:37,979 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:37,979 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:37,979 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:37,979 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:37,979 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:37,980 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85690 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:37,980 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:37,980 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/A is initiating minor compaction (all files) 2023-04-10 02:55:37,980 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/B is initiating minor compaction (all files) 2023-04-10 02:55:37,980 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/A in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,980 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/B in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:37,980 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/521fae27b38541ec997ce5b807db4120, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/33a4a252f7c84aeba58a36785625ef3e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/9671db15da2341dc83a2dc53e4abeb26] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=83.7 K 2023-04-10 02:55:37,980 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/abba928c353f49a58dbcba78dd3c0b0c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/c209ee994b5448f5875b349f7f268b0c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/5f3ef065621548d797da3969ab747299] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=36.2 K 2023-04-10 02:55:37,980 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 521fae27b38541ec997ce5b807db4120, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681095334601 2023-04-10 02:55:37,981 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting abba928c353f49a58dbcba78dd3c0b0c, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681095334601 2023-04-10 02:55:37,981 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 33a4a252f7c84aeba58a36785625ef3e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681095335223 2023-04-10 02:55:37,981 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting c209ee994b5448f5875b349f7f268b0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681095335223 2023-04-10 02:55:37,981 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 9671db15da2341dc83a2dc53e4abeb26, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681095336482 2023-04-10 02:55:37,981 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 5f3ef065621548d797da3969ab747299, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681095336482 2023-04-10 02:55:37,989 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#B#compaction#363 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:37,993 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#A#compaction#362 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:38,000 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/b5782b8c3b514b76a686979bac034976 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/b5782b8c3b514b76a686979bac034976 2023-04-10 02:55:38,005 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/B of f91f1c15c884b2146b2b8d0ffac5d524 into b5782b8c3b514b76a686979bac034976(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:38,005 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:38,005 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/B, priority=13, startTime=1681095337979; duration=0sec 2023-04-10 02:55:38,005 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:38,005 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:38,006 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:38,006 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/C is initiating minor compaction (all files) 2023-04-10 02:55:38,006 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/C in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:38,007 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/3ae4cf2f037d4f38a08bd15e0c2c59cd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/59a0606c07684d8c9ce73d08ce673906, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/cd6a8d0a721a4b97af7bc1d3d33d7045] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=36.2 K 2023-04-10 02:55:38,007 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 3ae4cf2f037d4f38a08bd15e0c2c59cd, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681095334601 2023-04-10 02:55:38,008 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 59a0606c07684d8c9ce73d08ce673906, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681095335223 2023-04-10 02:55:38,008 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting cd6a8d0a721a4b97af7bc1d3d33d7045, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681095336482 2023-04-10 02:55:38,025 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#C#compaction#364 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:38,056 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:38,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:38,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:38,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:38,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:38,059 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:38,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:38,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:38,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:38,061 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:38,061 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:38,062 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:38,062 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:38,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:38,062 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:38,062 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:38,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:38,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:38,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:38,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:38,064 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:38,064 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:38,064 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:38,064 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:38,064 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:38,064 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:38,064 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:38,065 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-10 02:55:38,065 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:38,065 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:38,065 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:38,065 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:38,065 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:38,065 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:38,078 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410fb0eb47e8cc749d8bebc09b2630b9a92 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fb0eb47e8cc749d8bebc09b2630b9a92 2023-04-10 02:55:38,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:38,263 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:38,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095338300, max timestamp=1681095338314Num uniques 859; 2023-04-10 02:55:38,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:55:38,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095338300, max timestamp=1681095338314Num uniques 859; 2023-04-10 02:55:38,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:55:38,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095338300, max timestamp=1681095338314Num uniques 859; 2023-04-10 02:55:38,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:55:38,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681095338348, max timestamp=1681095338361Num uniques 1018; 2023-04-10 02:55:38,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681095338348, max timestamp=1681095338361Num uniques 1018; 2023-04-10 02:55:38,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681095338348, max timestamp=1681095338361Num uniques 1018; 2023-04-10 02:55:38,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-10 02:55:38,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-10 02:55:38,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-10 02:55:38,422 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/68f10be70a8241bdbe7b946b37b56f8d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/68f10be70a8241bdbe7b946b37b56f8d 2023-04-10 02:55:38,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681095338379, max timestamp=1681095338424Num uniques 1077; 2023-04-10 02:55:38,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681095338379, max timestamp=1681095338424Num uniques 1077; 2023-04-10 02:55:38,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-10 02:55:38,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681095338379, max timestamp=1681095338424Num uniques 1077; 2023-04-10 02:55:38,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-10 02:55:38,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-10 02:55:38,429 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/A of f91f1c15c884b2146b2b8d0ffac5d524 into 68f10be70a8241bdbe7b946b37b56f8d(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:38,429 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:38,429 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/A, priority=13, startTime=1681095337979; duration=0sec 2023-04-10 02:55:38,429 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:38,448 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/86ee7871028e4527b4a851b728d2f57f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/86ee7871028e4527b4a851b728d2f57f 2023-04-10 02:55:38,464 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/C of f91f1c15c884b2146b2b8d0ffac5d524 into 86ee7871028e4527b4a851b728d2f57f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:38,464 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:38,464 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/C, priority=13, startTime=1681095337979; duration=0sec 2023-04-10 02:55:38,464 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:38,484 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=352, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/eb14589d2ba34f508ce5a5aee5557364 2023-04-10 02:55:38,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:38,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095398489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:38,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:38,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095398493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:38,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:38,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095398597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:38,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:38,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095398597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:38,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:38,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095398800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:38,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:38,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095398801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:38,892 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/ada93d1106924265bd5ef1b093c638a8 2023-04-10 02:55:39,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:39,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095399104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:39,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095399104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,303 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/a7afcb4d1618487c98ab7b01555f6415 2023-04-10 02:55:39,307 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/eb14589d2ba34f508ce5a5aee5557364 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/eb14589d2ba34f508ce5a5aee5557364 2023-04-10 02:55:39,310 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/eb14589d2ba34f508ce5a5aee5557364, entries=150, sequenceid=352, filesize=25.4 K 2023-04-10 02:55:39,310 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/ada93d1106924265bd5ef1b093c638a8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/ada93d1106924265bd5ef1b093c638a8 2023-04-10 02:55:39,314 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/ada93d1106924265bd5ef1b093c638a8, entries=150, sequenceid=352, filesize=11.9 K 2023-04-10 02:55:39,318 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/a7afcb4d1618487c98ab7b01555f6415 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/a7afcb4d1618487c98ab7b01555f6415 2023-04-10 02:55:39,321 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/a7afcb4d1618487c98ab7b01555f6415, entries=150, sequenceid=352, filesize=11.9 K 2023-04-10 02:55:39,322 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for f91f1c15c884b2146b2b8d0ffac5d524 in 1257ms, sequenceid=352, compaction requested=false 2023-04-10 02:55:39,322 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:39,322 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:39,322 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:39,323 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:39,323 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,323 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:39,323 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:39,324 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:39,324 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:39,324 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:39,324 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:39,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:39,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@94d32d1[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:39,326 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:39,326 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:39,328 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:39,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:39,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:39,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:39,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:39,328 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,328 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:39,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:39,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:39,329 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:39,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:39,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:39,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:39,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:39,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:39,332 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:39,332 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:39,332 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:39,332 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:39,332 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:39,334 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:39,334 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:39,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:39,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:39,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:39,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:39,334 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:39,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:39,334 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:39,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:39,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:39,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:39,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:39,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:39,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:39,336 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:39,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:39,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,338 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:39,338 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:39,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:39,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:39,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:39,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:39,338 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:39,338 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,339 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:39,339 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:39,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:39,339 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:39,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:39,339 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:39,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:39,339 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,339 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:39,339 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:39,339 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:39,439 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:39,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:39,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:39,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:39,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:39,442 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:39,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:39,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:39,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:39,442 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:39,443 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:39,446 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,446 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:39,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:39,447 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,447 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:39,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:39,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:39,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:39,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:39,448 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:39,448 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:39,448 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:39,448 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:39,448 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:39,448 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:39,448 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:39,449 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=147.60 KB heapSize=413.77 KB 2023-04-10 02:55:39,449 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:39,449 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:39,449 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:39,449 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:39,449 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:39,449 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:39,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:39,617 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:39,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:39,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095399625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:39,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095399626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:39,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095399731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:39,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095399731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,860 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410cef6bf34b1ca4942a5fbed8f78d69a71 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410cef6bf34b1ca4942a5fbed8f78d69a71 2023-04-10 02:55:39,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:39,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095399937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:39,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:39,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095399938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:40,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095400241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:40,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095400245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,265 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=425, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/a86a868e322449c99454e1dd491faf70 2023-04-10 02:55:40,677 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/74d6cb401e334f288c7ecfa205a836c1 2023-04-10 02:55:40,701 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/49dce2fb4fa54fac93ffa8478ed89520 2023-04-10 02:55:40,705 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/a86a868e322449c99454e1dd491faf70 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/a86a868e322449c99454e1dd491faf70 2023-04-10 02:55:40,708 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/a86a868e322449c99454e1dd491faf70, entries=150, sequenceid=425, filesize=25.4 K 2023-04-10 02:55:40,709 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/74d6cb401e334f288c7ecfa205a836c1 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/74d6cb401e334f288c7ecfa205a836c1 2023-04-10 02:55:40,712 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/74d6cb401e334f288c7ecfa205a836c1, entries=150, sequenceid=425, filesize=11.9 K 2023-04-10 02:55:40,713 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/49dce2fb4fa54fac93ffa8478ed89520 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/49dce2fb4fa54fac93ffa8478ed89520 2023-04-10 02:55:40,716 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/49dce2fb4fa54fac93ffa8478ed89520, entries=150, sequenceid=425, filesize=11.9 K 2023-04-10 02:55:40,716 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~413.72 KB/423648, currentSize=33.54 KB/34350 for f91f1c15c884b2146b2b8d0ffac5d524 in 1268ms, sequenceid=425, compaction requested=true 2023-04-10 02:55:40,716 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:40,716 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:40,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:40,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:40,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:40,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:40,719 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,719 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:40,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:40,719 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,719 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:40,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:40,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:40,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:40,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:40,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69d73e65[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:40,722 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:40,722 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,723 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,724 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:40,724 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:40,724 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:40,724 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,724 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:40,726 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:40,726 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:40,726 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:40,727 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:40,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:40,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:40,730 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:40,730 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:40,730 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:40,730 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:40,730 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:40,732 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:40,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:40,733 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,733 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:40,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:40,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:40,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:40,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:40,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:40,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:40,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:40,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:40,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:40,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:40,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:40,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,739 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:40,739 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:40,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:40,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:40,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:40,757 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:55:40,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:40,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:40,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:40,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:40,758 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:40,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:40,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095340792, max timestamp=1681095340803Num uniques 859; 2023-04-10 02:55:40,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:55:40,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095340792, max timestamp=1681095340803Num uniques 859; 2023-04-10 02:55:40,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095340792, max timestamp=1681095340803Num uniques 859; 2023-04-10 02:55:40,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:40,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:40,839 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:40,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:40,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:40,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:40,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:40,843 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:40,843 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:40,843 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:40,843 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:40,843 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,843 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:40,844 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,844 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:40,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:40,845 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,845 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:40,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:40,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:40,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:40,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:40,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:40,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:40,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:40,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:40,846 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:40,846 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:40,846 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:40,846 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:40,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:40,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:40,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:40,846 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:40,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:40,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095400841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,848 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,848 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:40,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:40,848 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,848 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:40,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:40,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:40,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:40,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095400843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:40,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:40,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b76f548[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:40,850 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:40,850 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,856 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,856 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:40,856 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:40,856 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:40,856 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,856 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:40,858 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:40,859 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:40,859 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:40,859 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:40,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:40,860 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:40,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:40,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:40,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:40,860 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:40,862 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,862 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:40,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,862 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:40,862 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,863 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,874 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,874 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:40,874 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:40,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:40,874 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:40,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:40,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:40,874 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:40,874 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:40,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:40,875 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:40,875 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:40,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:40,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,875 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:40,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095400947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:40,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095400950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,975 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:40,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:40,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:40,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:40,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:40,978 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:40,978 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:40,978 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:40,978 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:40,979 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,979 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:40,980 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,980 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:40,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:40,980 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,980 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:40,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:40,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:40,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:40,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:40,981 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:40,981 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:40,981 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:40,982 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:40,982 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:40,982 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:40,982 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:40,982 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:40,982 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:40,982 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:40,982 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:40,982 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,982 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:40,982 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:40,983 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,983 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:40,984 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,984 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:40,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:40,985 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:40,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@716e4fce[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:40,985 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,986 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,986 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:40,986 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:40,986 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:40,987 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,987 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:40,988 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,988 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:40,989 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:40,989 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:40,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:40,992 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:40,992 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:40,992 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:40,992 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:40,992 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:40,994 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,994 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,994 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:40,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:40,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:40,994 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,995 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:40,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:40,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,997 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,997 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:40,998 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:40,998 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:40,998 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:40,998 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:40,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:40,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:40,998 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:40,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:40,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:40,998 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,998 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:40,998 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:40,998 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:40,998 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:40,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:40,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,098 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:41,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:41,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:41,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:41,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:41,101 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:41,102 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,102 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:41,102 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:41,102 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,102 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,103 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,103 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:41,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:41,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:41,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:41,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:41,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:41,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:41,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:41,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:41,105 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,105 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:41,105 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:41,105 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:41,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:41,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:41,106 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:41,108 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11552088[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:41,109 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:41,110 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,111 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,111 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:41,111 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:41,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:41,111 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:41,114 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:41,114 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,114 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,114 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:41,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:41,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:41,116 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:41,116 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:41,116 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:41,116 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:41,116 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:41,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,118 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,119 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:41,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:41,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,124 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,124 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:41,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,124 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:41,125 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,125 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,125 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:41,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095401155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:41,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095401155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,169 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410eacf2cf2538d4b07b5f7245ff47422b9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410eacf2cf2538d4b07b5f7245ff47422b9 2023-04-10 02:55:41,225 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:41,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:41,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:41,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:41,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:41,228 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:41,228 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,228 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:41,228 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:41,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,229 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,230 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,230 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,230 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,230 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:41,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:41,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:41,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:41,231 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:41,231 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,231 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:41,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:41,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:41,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:41,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:41,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:41,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:41,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:41,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:41,233 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,233 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,234 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,234 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:41,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:41,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41a100d1[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:41,235 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:41,235 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,238 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:41,238 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,238 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:41,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:41,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:41,240 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,241 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:41,241 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,241 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:41,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:41,243 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:41,243 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:41,243 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:41,243 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:41,243 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:41,244 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,244 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,244 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,245 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,246 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,251 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,251 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,251 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,251 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,251 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:41,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:41,251 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,252 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,252 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,252 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,252 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:41,252 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:41,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,352 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:41,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:41,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:41,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:41,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:41,355 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:41,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:41,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:41,356 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,356 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,357 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,357 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,358 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,358 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:41,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:41,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:41,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:41,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:41,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:41,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:41,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:41,359 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,359 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:41,359 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:41,359 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:41,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:41,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:41,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:41,361 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,361 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,361 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,361 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:41,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1da16118[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:41,363 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:41,363 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,364 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,364 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:41,364 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:41,364 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:41,364 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,364 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:41,365 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,365 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:41,365 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,365 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:41,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:41,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:41,367 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:41,367 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:41,367 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:41,367 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:41,368 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:41,370 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,370 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,370 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,370 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,374 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,374 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,374 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,374 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,374 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:41,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:41,374 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,374 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,374 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,375 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:41,375 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:41,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:41,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095401462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:41,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095401463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,475 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:41,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:41,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:41,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:41,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:41,478 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:41,478 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,478 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:41,478 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:41,479 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,479 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,480 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,480 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,480 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,480 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:41,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:41,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:41,481 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:41,481 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,481 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:41,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:41,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:41,482 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,482 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:41,482 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:41,482 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:41,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:41,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:41,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:41,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,484 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:41,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:41,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb25d5d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:41,486 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:41,486 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,488 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,489 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:41,489 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:41,489 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:41,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:41,490 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,490 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:41,490 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,490 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:41,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:41,494 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:41,494 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:41,494 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:41,494 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:41,494 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:41,496 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,496 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,496 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,500 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,500 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,500 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,500 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,501 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,501 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,501 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,501 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,501 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,501 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:41,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:41,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,502 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:41,502 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:41,574 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=436, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/878121de5dfa41b790db9e987944bce4 2023-04-10 02:55:41,602 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:41,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:41,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:41,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:41,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:41,605 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:41,605 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,605 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:41,605 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:41,605 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,605 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,607 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:41,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:41,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:41,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:41,609 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,609 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:41,609 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:41,609 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:41,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,611 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:41,613 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:41,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26db7f63[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:41,613 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,614 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,614 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:41,614 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:41,614 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:41,614 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,614 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:41,615 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,615 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:41,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,615 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:41,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:41,618 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:41,618 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:41,618 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:41,618 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:41,618 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:41,620 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,620 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,621 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,621 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,624 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,624 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,624 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,625 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,625 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,625 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,625 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,625 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,625 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:41,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:41,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:41,625 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,625 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:41,625 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:41,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,725 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:41,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:41,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:41,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:41,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:41,729 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:41,729 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,729 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:41,729 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:41,729 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,729 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,731 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,731 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,731 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,731 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:41,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:41,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:41,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:41,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:41,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:41,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:41,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:41,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:41,732 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,732 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:41,732 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:41,732 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:41,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:41,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,733 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:41,733 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:41,737 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,737 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,738 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,738 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:41,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:41,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f3544e1[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:41,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:41,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,740 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:41,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:41,740 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,740 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:41,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:41,742 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:41,742 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,742 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,742 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:41,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:41,744 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:41,744 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:41,744 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:41,744 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:41,744 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:41,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,746 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,747 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,748 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:41,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:41,751 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,751 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:41,751 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:41,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,851 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:41,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:41,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:41,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:41,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:41,854 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:41,854 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,854 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:41,854 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:41,855 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,855 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,856 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,856 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,856 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,856 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:41,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:41,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:41,857 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:41,857 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,857 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:41,858 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:41,858 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:41,858 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,858 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:41,858 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:41,858 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,858 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:41,858 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:41,858 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,858 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:41,858 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:41,859 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,859 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,860 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,860 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:41,861 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:41,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790a51be[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:41,861 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,862 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,862 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:41,862 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:41,862 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:41,862 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,863 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:41,864 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,864 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:41,864 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,864 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:41,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:41,867 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:41,867 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:41,867 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:41,867 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:41,867 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:41,868 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,868 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,868 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,869 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,870 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,872 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,872 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,872 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,872 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,872 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:41,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:41,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:41,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:41,872 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,872 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,872 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,872 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,872 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,872 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:41,872 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:41,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:41,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:41,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:41,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36704 deadline: 1681095401966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,973 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:41,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:41,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:41,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36650 deadline: 1681095401968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:41,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:41,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:41,978 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:41,978 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,978 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:41,978 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:41,979 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,979 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,980 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:41,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:41,980 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,981 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,981 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:41,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:41,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:41,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:41,982 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:41,982 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:41,982 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:41,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/8e3235d68b1c4ef9afcaedbf58eca85e 2023-04-10 02:55:41,984 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,984 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:41,984 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:41,984 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:41,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,991 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,991 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,991 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:41,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:41,993 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:41,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755eb377[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:41,993 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,995 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:41,995 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,995 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:41,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:41,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:41,995 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:41,996 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,996 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:41,996 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:41,996 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:41,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:41,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:41,999 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:41,999 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:41,999 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:41,999 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:41,999 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:42,001 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,001 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:42,001 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:42,001 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:42,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,003 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,006 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,006 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,006 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,006 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,006 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,006 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,006 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,006 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:42,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:42,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:42,006 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:42,006 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:42,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:42,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:42,007 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:42,007 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:42,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:42,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:42,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:42,107 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:42,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:42,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:42,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:42,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:42,110 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:42,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:42,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:42,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:42,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,112 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:42,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:42,112 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,113 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,113 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:42,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:42,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:42,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:42,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:42,114 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:42,114 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:42,114 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:42,114 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:42,114 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:42,115 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:42,115 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:42,115 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:42,115 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:42,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:42,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:42,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:42,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:42,116 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,116 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:42,117 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,117 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:42,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:42,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:42,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:42,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:42,118 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:42,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@803fad8[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:42,118 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,119 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,119 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:42,119 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:42,119 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:42,119 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,120 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:42,127 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:42,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:42,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:42,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:42,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:42,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:42,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:42,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:42,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:42,129 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:42,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:42,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:42,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:42,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:42,129 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:42,130 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,130 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,130 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:42,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:42,131 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,132 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:42,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:42,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:42,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:42,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:42,135 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,135 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,135 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,135 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:42,135 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:42,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:42,134 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:42,135 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:42,235 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:42,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:42,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:42,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:42,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:42,238 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:42,238 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:42,238 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:42,238 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:42,239 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,239 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,240 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:42,240 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:42,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:42,241 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,241 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:42,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:42,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:42,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:42,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:42,242 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:42,242 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:42,242 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:42,242 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:42,242 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:42,242 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:42,242 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:42,242 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:42,242 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:42,242 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:42,242 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:42,242 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,242 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:42,242 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:42,244 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,244 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:42,245 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,245 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:42,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:42,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d54a575[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:42,246 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:42,246 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,247 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:42,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:42,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:42,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:42,249 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,249 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:42,249 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:42,249 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:42,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:42,251 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:42,252 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:42,252 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:42,252 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:42,252 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:42,253 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,253 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,253 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:42,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:42,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:42,254 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,255 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:42,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:42,259 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:42,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:42,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:42,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:42,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:42,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:42,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:42,259 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:42,259 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:42,360 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:42,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:42,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:42,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:42,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:42,362 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:42,363 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:42,363 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:42,363 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:42,363 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,363 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,364 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,364 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:42,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:42,365 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,365 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:42,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:42,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:42,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:42,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:42,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:42,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:42,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:42,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:42,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:42,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:42,368 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:42,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:42,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:42,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:42,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:42,369 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,369 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:42,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:42,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:42,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f88ac0[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:42,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:42,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,372 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,372 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,373 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:42,372 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:42,373 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:42,373 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:42,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,375 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:42,375 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:42,375 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:42,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:42,377 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:42,377 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:42,377 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:42,377 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:42,377 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:42,378 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,378 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,378 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:42,379 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,380 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:42,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:42,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,382 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:42,382 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,382 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:42,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:42,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:42,382 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:42,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:42,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:42,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:42,382 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:42,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:42,383 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:42,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:42,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,383 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:42,394 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/4027af0d7287416fa564b2045c1be52b 2023-04-10 02:55:42,400 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/878121de5dfa41b790db9e987944bce4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/878121de5dfa41b790db9e987944bce4 2023-04-10 02:55:42,404 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/878121de5dfa41b790db9e987944bce4, entries=200, sequenceid=436, filesize=32.3 K 2023-04-10 02:55:42,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/8e3235d68b1c4ef9afcaedbf58eca85e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/8e3235d68b1c4ef9afcaedbf58eca85e 2023-04-10 02:55:42,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/8e3235d68b1c4ef9afcaedbf58eca85e, entries=150, sequenceid=436, filesize=11.9 K 2023-04-10 02:55:42,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/4027af0d7287416fa564b2045c1be52b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/4027af0d7287416fa564b2045c1be52b 2023-04-10 02:55:42,412 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/4027af0d7287416fa564b2045c1be52b, entries=150, sequenceid=436, filesize=11.9 K 2023-04-10 02:55:42,412 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for f91f1c15c884b2146b2b8d0ffac5d524 in 1655ms, sequenceid=436, compaction requested=true 2023-04-10 02:55:42,413 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:42,413 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:42,413 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:42,413 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:55:42,413 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:55:42,413 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:42,414 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:55:42,414 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/B is initiating minor compaction (all files) 2023-04-10 02:55:42,414 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/B in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:42,414 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/b5782b8c3b514b76a686979bac034976, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/ada93d1106924265bd5ef1b093c638a8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/74d6cb401e334f288c7ecfa205a836c1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/8e3235d68b1c4ef9afcaedbf58eca85e] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=48.2 K 2023-04-10 02:55:42,414 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting b5782b8c3b514b76a686979bac034976, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681095336482 2023-04-10 02:55:42,415 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting ada93d1106924265bd5ef1b093c638a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1681095337120 2023-04-10 02:55:42,415 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111838 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:55:42,415 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/A is initiating minor compaction (all files) 2023-04-10 02:55:42,415 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 74d6cb401e334f288c7ecfa205a836c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681095338487 2023-04-10 02:55:42,416 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/A in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:42,416 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/68f10be70a8241bdbe7b946b37b56f8d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/eb14589d2ba34f508ce5a5aee5557364, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/a86a868e322449c99454e1dd491faf70, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/878121de5dfa41b790db9e987944bce4] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=109.2 K 2023-04-10 02:55:42,416 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 8e3235d68b1c4ef9afcaedbf58eca85e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681095339623 2023-04-10 02:55:42,416 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 68f10be70a8241bdbe7b946b37b56f8d, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681095336482 2023-04-10 02:55:42,416 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting eb14589d2ba34f508ce5a5aee5557364, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1681095337120 2023-04-10 02:55:42,417 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting a86a868e322449c99454e1dd491faf70, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681095338487 2023-04-10 02:55:42,417 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 878121de5dfa41b790db9e987944bce4, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681095339623 2023-04-10 02:55:42,445 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#B#compaction#375 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:42,448 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#A#compaction#374 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:42,463 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/6243f04f856c455da2d8c5fd89506d3f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/6243f04f856c455da2d8c5fd89506d3f 2023-04-10 02:55:42,468 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/B of f91f1c15c884b2146b2b8d0ffac5d524 into 6243f04f856c455da2d8c5fd89506d3f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:42,468 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:42,468 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/B, priority=12, startTime=1681095342413; duration=0sec 2023-04-10 02:55:42,468 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:42,468 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:55:42,469 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:55:42,470 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): f91f1c15c884b2146b2b8d0ffac5d524/C is initiating minor compaction (all files) 2023-04-10 02:55:42,470 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f91f1c15c884b2146b2b8d0ffac5d524/C in TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:42,470 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/86ee7871028e4527b4a851b728d2f57f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/a7afcb4d1618487c98ab7b01555f6415, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/49dce2fb4fa54fac93ffa8478ed89520, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/4027af0d7287416fa564b2045c1be52b] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp, totalSize=48.2 K 2023-04-10 02:55:42,470 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 86ee7871028e4527b4a851b728d2f57f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681095336482 2023-04-10 02:55:42,471 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting a7afcb4d1618487c98ab7b01555f6415, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1681095337120 2023-04-10 02:55:42,471 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 49dce2fb4fa54fac93ffa8478ed89520, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681095338487 2023-04-10 02:55:42,471 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 4027af0d7287416fa564b2045c1be52b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681095339623 2023-04-10 02:55:42,479 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): f91f1c15c884b2146b2b8d0ffac5d524#C#compaction#376 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:42,483 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/1b06dd5d959e490dbf586dfde6fbe3db as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/1b06dd5d959e490dbf586dfde6fbe3db 2023-04-10 02:55:42,483 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:42,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:42,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:42,488 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/A of f91f1c15c884b2146b2b8d0ffac5d524 into 1b06dd5d959e490dbf586dfde6fbe3db(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:42,488 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:42,488 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/A, priority=12, startTime=1681095342413; duration=0sec 2023-04-10 02:55:42,488 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:42,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:42,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:42,489 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:42,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:42,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:42,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:42,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,491 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:42,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:42,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:42,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:42,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:42,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:42,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:42,492 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:42,492 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:42,492 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:42,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:42,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:42,495 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:42,495 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. started... 2023-04-10 02:55:42,495 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-10 02:55:42,496 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:42,496 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:42,496 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:42,496 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:42,496 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:42,496 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:42,497 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/b9179769db654664af4d012e8f76c3c9 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/b9179769db654664af4d012e8f76c3c9 2023-04-10 02:55:42,500 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f91f1c15c884b2146b2b8d0ffac5d524/C of f91f1c15c884b2146b2b8d0ffac5d524 into b9179769db654664af4d012e8f76c3c9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:42,500 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:42,500 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524., storeName=f91f1c15c884b2146b2b8d0ffac5d524/C, priority=12, startTime=1681095342413; duration=0sec 2023-04-10 02:55:42,501 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:42,907 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410fe984059ccf9430dac8c705694945685 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fe984059ccf9430dac8c705694945685 2023-04-10 02:55:42,947 DEBUG [Thread-2321] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a7503d4 to 127.0.0.1:60568 2023-04-10 02:55:42,948 DEBUG [Thread-2321] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:42,948 DEBUG [Thread-2317] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e86d9 to 127.0.0.1:60568 2023-04-10 02:55:42,948 DEBUG [Thread-2317] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:42,952 DEBUG [Thread-2319] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x711ebaf2 to 127.0.0.1:60568 2023-04-10 02:55:42,952 DEBUG [Thread-2319] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:42,952 DEBUG [Thread-2313] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10ee5f06 to 127.0.0.1:60568 2023-04-10 02:55:42,952 DEBUG [Thread-2313] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:42,955 DEBUG [Thread-2315] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21128196 to 127.0.0.1:60568 2023-04-10 02:55:42,955 DEBUG [Thread-2315] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:42,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:42,975 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. as already flushing 2023-04-10 02:55:42,975 DEBUG [Thread-2308] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69100738 to 127.0.0.1:60568 2023-04-10 02:55:42,975 DEBUG [Thread-2308] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:42,985 DEBUG [Thread-2310] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f603c27 to 127.0.0.1:60568 2023-04-10 02:55:42,985 DEBUG [Thread-2310] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:43,331 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=473, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/581583e827554713bfc4019866768593 2023-04-10 02:55:43,510 DEBUG [Thread-2304] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x445919fd to 127.0.0.1:60568 2023-04-10 02:55:43,510 DEBUG [Thread-2304] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:43,527 DEBUG [Thread-2302] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bb27fd4 to 127.0.0.1:60568 2023-04-10 02:55:43,527 DEBUG [Thread-2302] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:43,567 DEBUG [Thread-2306] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d010713 to 127.0.0.1:60568 2023-04-10 02:55:43,567 DEBUG [Thread-2306] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:43,739 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/7fbfb2c3b65c4089883bde70e5be8822 2023-04-10 02:55:44,148 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/b2fc7ee8d395475fbe00295a9a42b69e 2023-04-10 02:55:44,151 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/581583e827554713bfc4019866768593 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/581583e827554713bfc4019866768593 2023-04-10 02:55:44,154 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/581583e827554713bfc4019866768593, entries=150, sequenceid=473, filesize=25.4 K 2023-04-10 02:55:44,154 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/7fbfb2c3b65c4089883bde70e5be8822 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/7fbfb2c3b65c4089883bde70e5be8822 2023-04-10 02:55:44,157 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/7fbfb2c3b65c4089883bde70e5be8822, entries=150, sequenceid=473, filesize=11.9 K 2023-04-10 02:55:44,157 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/b2fc7ee8d395475fbe00295a9a42b69e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/b2fc7ee8d395475fbe00295a9a42b69e 2023-04-10 02:55:44,160 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/b2fc7ee8d395475fbe00295a9a42b69e, entries=150, sequenceid=473, filesize=11.9 K 2023-04-10 02:55:44,161 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=33.54 KB/34350 for f91f1c15c884b2146b2b8d0ffac5d524 in 1666ms, sequenceid=473, compaction requested=false 2023-04-10 02:55:44,161 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:44,161 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:44,161 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:44,161 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:44,161 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,161 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:44,161 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:44,165 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,166 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:44,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:44,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:44,166 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:44,166 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:44,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:44,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:44,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:44,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:44,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:44,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66bd4863[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:44,167 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:44,167 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:44,169 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:44,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:44,169 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:44,169 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:44,169 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:44,169 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,169 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:44,170 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,170 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:44,170 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:44,170 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:44,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:44,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:44,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:44,172 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:44,172 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:44,172 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:44,172 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:44,172 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:44,174 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:44,174 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:44,174 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:44,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:44,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:44,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:44,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:44,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:44,174 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:44,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:44,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:44,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:44,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:44,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:44,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:44,175 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:44,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:44,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,177 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,177 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:44,177 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:44,177 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:44,177 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:44,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:44,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:44,177 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:44,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:44,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:44,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:44,177 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,177 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:44,177 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:44,177 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:44,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:44,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:44,177 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:44,177 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:44,277 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:44,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:44,278 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-10 02:55:44,278 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 18 2023-04-10 02:55:44,278 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 17 2023-04-10 02:55:44,278 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 27 2023-04-10 02:55:44,278 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 182 2023-04-10 02:55:44,278 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 159 2023-04-10 02:55:44,278 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-10 02:55:44,279 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-10 02:55:44,279 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 1774 2023-04-10 02:55:44,279 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 5322 rows 2023-04-10 02:55:44,279 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 1770 2023-04-10 02:55:44,279 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 5309 rows 2023-04-10 02:55:44,279 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 1778 2023-04-10 02:55:44,279 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 5334 rows 2023-04-10 02:55:44,279 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 1773 2023-04-10 02:55:44,279 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 5315 rows 2023-04-10 02:55:44,279 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 1775 2023-04-10 02:55:44,279 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 5321 rows 2023-04-10 02:55:44,279 INFO [Listener at localhost/44533] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-10 02:55:44,279 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4473f45c to 127.0.0.1:60568 2023-04-10 02:55:44,279 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:55:44,280 INFO [Listener at localhost/44533] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-10 02:55:44,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-10 02:55:44,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:44,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-10 02:55:44,286 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095344286"}]},"ts":"1681095344286"} 2023-04-10 02:55:44,287 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-10 02:55:44,288 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-10 02:55:44,289 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f91f1c15c884b2146b2b8d0ffac5d524, UNASSIGN}] 2023-04-10 02:55:44,290 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f91f1c15c884b2146b2b8d0ffac5d524, UNASSIGN 2023-04-10 02:55:44,291 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-10 02:55:44,291 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=f91f1c15c884b2146b2b8d0ffac5d524, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:44,292 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095344291"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095344291"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095344291"}]},"ts":"1681095344291"} 2023-04-10 02:55:44,298 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:55:44,318 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/9d30ac409d314f1d84f10445cefd5b78, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/eebf4c95b68548eb9b77cba44468464c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/5b0a81761ed94076a4f47d24a8889de4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/347c651f9b5544d29c5b2ca52056036d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/543f3193a37d4c64a8dfe76037a3ea52, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/7a8fcf0d20f345dd91a574be55db0ae9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/85eaaf96e11b4253bc9dcea70de49eab, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/0f18612cceee460c9037656a5b4fceb9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/e70cfc927c6443d7a1ea38654f9590e1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/ddf88e6392fc435d895c1db43b3b87cb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/00fb516cc3d144f7a9282ca1cf5d2a81, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/b1d44c6e244e4818bd398bb4c409c9c0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/7e3de2cd68da40b2b85629eb314e7137, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/c180bd081ed34487a3e81343e7e38282, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/5e512f0191704438bf32156f58ea4a7d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/521fae27b38541ec997ce5b807db4120, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/33a4a252f7c84aeba58a36785625ef3e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/9671db15da2341dc83a2dc53e4abeb26, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/68f10be70a8241bdbe7b946b37b56f8d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/eb14589d2ba34f508ce5a5aee5557364, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/a86a868e322449c99454e1dd491faf70, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/878121de5dfa41b790db9e987944bce4] to archive 2023-04-10 02:55:44,319 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:55:44,320 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/9d30ac409d314f1d84f10445cefd5b78 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/9d30ac409d314f1d84f10445cefd5b78 2023-04-10 02:55:44,321 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/eebf4c95b68548eb9b77cba44468464c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/eebf4c95b68548eb9b77cba44468464c 2023-04-10 02:55:44,322 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/5b0a81761ed94076a4f47d24a8889de4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/5b0a81761ed94076a4f47d24a8889de4 2023-04-10 02:55:44,323 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/347c651f9b5544d29c5b2ca52056036d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/347c651f9b5544d29c5b2ca52056036d 2023-04-10 02:55:44,325 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/543f3193a37d4c64a8dfe76037a3ea52 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/543f3193a37d4c64a8dfe76037a3ea52 2023-04-10 02:55:44,325 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/7a8fcf0d20f345dd91a574be55db0ae9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/7a8fcf0d20f345dd91a574be55db0ae9 2023-04-10 02:55:44,327 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/85eaaf96e11b4253bc9dcea70de49eab to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/85eaaf96e11b4253bc9dcea70de49eab 2023-04-10 02:55:44,328 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/0f18612cceee460c9037656a5b4fceb9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/0f18612cceee460c9037656a5b4fceb9 2023-04-10 02:55:44,328 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/e70cfc927c6443d7a1ea38654f9590e1 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/e70cfc927c6443d7a1ea38654f9590e1 2023-04-10 02:55:44,329 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/ddf88e6392fc435d895c1db43b3b87cb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/ddf88e6392fc435d895c1db43b3b87cb 2023-04-10 02:55:44,330 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/00fb516cc3d144f7a9282ca1cf5d2a81 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/00fb516cc3d144f7a9282ca1cf5d2a81 2023-04-10 02:55:44,331 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/b1d44c6e244e4818bd398bb4c409c9c0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/b1d44c6e244e4818bd398bb4c409c9c0 2023-04-10 02:55:44,332 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/7e3de2cd68da40b2b85629eb314e7137 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/7e3de2cd68da40b2b85629eb314e7137 2023-04-10 02:55:44,333 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/c180bd081ed34487a3e81343e7e38282 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/c180bd081ed34487a3e81343e7e38282 2023-04-10 02:55:44,333 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/5e512f0191704438bf32156f58ea4a7d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/5e512f0191704438bf32156f58ea4a7d 2023-04-10 02:55:44,334 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/521fae27b38541ec997ce5b807db4120 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/521fae27b38541ec997ce5b807db4120 2023-04-10 02:55:44,335 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/33a4a252f7c84aeba58a36785625ef3e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/33a4a252f7c84aeba58a36785625ef3e 2023-04-10 02:55:44,335 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/9671db15da2341dc83a2dc53e4abeb26 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/9671db15da2341dc83a2dc53e4abeb26 2023-04-10 02:55:44,336 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/68f10be70a8241bdbe7b946b37b56f8d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/68f10be70a8241bdbe7b946b37b56f8d 2023-04-10 02:55:44,337 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/eb14589d2ba34f508ce5a5aee5557364 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/eb14589d2ba34f508ce5a5aee5557364 2023-04-10 02:55:44,337 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/a86a868e322449c99454e1dd491faf70 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/a86a868e322449c99454e1dd491faf70 2023-04-10 02:55:44,338 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/878121de5dfa41b790db9e987944bce4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/878121de5dfa41b790db9e987944bce4 2023-04-10 02:55:44,342 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/9759e005628b43bb94efd726d43744a3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/97ee68bb77c84805a0cc90af0f214fa3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/6ca4706c30244fbe989c279156c73349, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/a3bc55160d154b958ba683b6c874f64c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/5066d8615b224398bca9f6f950a696f4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/e87d0eff2f574e9fb313fcbc3578904b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/f69e5198878f4887afccc84d08abd2b0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/1491c7e30dac46e0abb26e5e7b65e7a3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/1814305a929a4f74bd06685357a4d162, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/ab980306661946e0a5622146e3e421f1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/28eae182836d400384376f6c8ad0fe2f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/08060eb67abd49e08cb150146dacdb48, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/3d6d5c711cfe47cf83132cd75b4788cf, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/07400ba0fd4044c1bf44708950313a53, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/abba928c353f49a58dbcba78dd3c0b0c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/e3c423518f7d4916aa68303fc9da2a49, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/c209ee994b5448f5875b349f7f268b0c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/b5782b8c3b514b76a686979bac034976, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/5f3ef065621548d797da3969ab747299, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/ada93d1106924265bd5ef1b093c638a8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/74d6cb401e334f288c7ecfa205a836c1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/8e3235d68b1c4ef9afcaedbf58eca85e] to archive 2023-04-10 02:55:44,343 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:55:44,344 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/9759e005628b43bb94efd726d43744a3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/9759e005628b43bb94efd726d43744a3 2023-04-10 02:55:44,344 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/97ee68bb77c84805a0cc90af0f214fa3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/97ee68bb77c84805a0cc90af0f214fa3 2023-04-10 02:55:44,345 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/6ca4706c30244fbe989c279156c73349 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/6ca4706c30244fbe989c279156c73349 2023-04-10 02:55:44,346 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/a3bc55160d154b958ba683b6c874f64c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/a3bc55160d154b958ba683b6c874f64c 2023-04-10 02:55:44,347 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/5066d8615b224398bca9f6f950a696f4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/5066d8615b224398bca9f6f950a696f4 2023-04-10 02:55:44,348 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/e87d0eff2f574e9fb313fcbc3578904b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/e87d0eff2f574e9fb313fcbc3578904b 2023-04-10 02:55:44,348 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/f69e5198878f4887afccc84d08abd2b0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/f69e5198878f4887afccc84d08abd2b0 2023-04-10 02:55:44,349 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/1491c7e30dac46e0abb26e5e7b65e7a3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/1491c7e30dac46e0abb26e5e7b65e7a3 2023-04-10 02:55:44,350 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/1814305a929a4f74bd06685357a4d162 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/1814305a929a4f74bd06685357a4d162 2023-04-10 02:55:44,351 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/ab980306661946e0a5622146e3e421f1 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/ab980306661946e0a5622146e3e421f1 2023-04-10 02:55:44,351 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/28eae182836d400384376f6c8ad0fe2f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/28eae182836d400384376f6c8ad0fe2f 2023-04-10 02:55:44,352 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/08060eb67abd49e08cb150146dacdb48 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/08060eb67abd49e08cb150146dacdb48 2023-04-10 02:55:44,353 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/3d6d5c711cfe47cf83132cd75b4788cf to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/3d6d5c711cfe47cf83132cd75b4788cf 2023-04-10 02:55:44,353 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/07400ba0fd4044c1bf44708950313a53 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/07400ba0fd4044c1bf44708950313a53 2023-04-10 02:55:44,354 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/abba928c353f49a58dbcba78dd3c0b0c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/abba928c353f49a58dbcba78dd3c0b0c 2023-04-10 02:55:44,355 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/e3c423518f7d4916aa68303fc9da2a49 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/e3c423518f7d4916aa68303fc9da2a49 2023-04-10 02:55:44,356 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/c209ee994b5448f5875b349f7f268b0c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/c209ee994b5448f5875b349f7f268b0c 2023-04-10 02:55:44,357 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/b5782b8c3b514b76a686979bac034976 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/b5782b8c3b514b76a686979bac034976 2023-04-10 02:55:44,357 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/5f3ef065621548d797da3969ab747299 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/5f3ef065621548d797da3969ab747299 2023-04-10 02:55:44,358 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/ada93d1106924265bd5ef1b093c638a8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/ada93d1106924265bd5ef1b093c638a8 2023-04-10 02:55:44,359 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/74d6cb401e334f288c7ecfa205a836c1 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/74d6cb401e334f288c7ecfa205a836c1 2023-04-10 02:55:44,360 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/8e3235d68b1c4ef9afcaedbf58eca85e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/8e3235d68b1c4ef9afcaedbf58eca85e 2023-04-10 02:55:44,365 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/0d45321fa98c4724b086a676fd37caec, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/1dc666c6964843b2813e263cb89a742e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/3642b6f02e0c40c3b00b9d62fcd4c69e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/1a89c361285f41d8a3d1fffaa14b6bfb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/48d3a3ada45a4f7295a01323d6fb7067, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe503a2b82d543d2bea92aa077417e85, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/82ff01e523fb43eaaa781b75c3890603, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/76ebaf4be0c84d58b8649d8fb3ef2ae4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/6fda50453ab04466a2f128a0624811ea, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/b2b1cccca4d44883a1ce5c123f79a2bf, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/dd11c398496b422693c8184b07f6387a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe1d07ca99434fda9ef25a3b4614b48d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/685b63b5f0ea4546867ee37fd7789a41, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe8e35916aef4ee9800d4b104387ace6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/3ae4cf2f037d4f38a08bd15e0c2c59cd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/41a3c24c18514ca188106950efd5166f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/59a0606c07684d8c9ce73d08ce673906, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/86ee7871028e4527b4a851b728d2f57f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/cd6a8d0a721a4b97af7bc1d3d33d7045, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/a7afcb4d1618487c98ab7b01555f6415, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/49dce2fb4fa54fac93ffa8478ed89520, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/4027af0d7287416fa564b2045c1be52b] to archive 2023-04-10 02:55:44,366 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:55:44,367 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/0d45321fa98c4724b086a676fd37caec to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/0d45321fa98c4724b086a676fd37caec 2023-04-10 02:55:44,368 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/1dc666c6964843b2813e263cb89a742e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/1dc666c6964843b2813e263cb89a742e 2023-04-10 02:55:44,369 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/3642b6f02e0c40c3b00b9d62fcd4c69e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/3642b6f02e0c40c3b00b9d62fcd4c69e 2023-04-10 02:55:44,369 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/1a89c361285f41d8a3d1fffaa14b6bfb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/1a89c361285f41d8a3d1fffaa14b6bfb 2023-04-10 02:55:44,370 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/48d3a3ada45a4f7295a01323d6fb7067 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/48d3a3ada45a4f7295a01323d6fb7067 2023-04-10 02:55:44,371 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe503a2b82d543d2bea92aa077417e85 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe503a2b82d543d2bea92aa077417e85 2023-04-10 02:55:44,372 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/82ff01e523fb43eaaa781b75c3890603 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/82ff01e523fb43eaaa781b75c3890603 2023-04-10 02:55:44,373 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/76ebaf4be0c84d58b8649d8fb3ef2ae4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/76ebaf4be0c84d58b8649d8fb3ef2ae4 2023-04-10 02:55:44,374 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/6fda50453ab04466a2f128a0624811ea to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/6fda50453ab04466a2f128a0624811ea 2023-04-10 02:55:44,374 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/b2b1cccca4d44883a1ce5c123f79a2bf to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/b2b1cccca4d44883a1ce5c123f79a2bf 2023-04-10 02:55:44,375 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/dd11c398496b422693c8184b07f6387a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/dd11c398496b422693c8184b07f6387a 2023-04-10 02:55:44,376 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe1d07ca99434fda9ef25a3b4614b48d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe1d07ca99434fda9ef25a3b4614b48d 2023-04-10 02:55:44,377 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/685b63b5f0ea4546867ee37fd7789a41 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/685b63b5f0ea4546867ee37fd7789a41 2023-04-10 02:55:44,378 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe8e35916aef4ee9800d4b104387ace6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/fe8e35916aef4ee9800d4b104387ace6 2023-04-10 02:55:44,379 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/3ae4cf2f037d4f38a08bd15e0c2c59cd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/3ae4cf2f037d4f38a08bd15e0c2c59cd 2023-04-10 02:55:44,379 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/41a3c24c18514ca188106950efd5166f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/41a3c24c18514ca188106950efd5166f 2023-04-10 02:55:44,380 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/59a0606c07684d8c9ce73d08ce673906 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/59a0606c07684d8c9ce73d08ce673906 2023-04-10 02:55:44,381 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/86ee7871028e4527b4a851b728d2f57f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/86ee7871028e4527b4a851b728d2f57f 2023-04-10 02:55:44,382 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/cd6a8d0a721a4b97af7bc1d3d33d7045 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/cd6a8d0a721a4b97af7bc1d3d33d7045 2023-04-10 02:55:44,383 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/a7afcb4d1618487c98ab7b01555f6415 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/a7afcb4d1618487c98ab7b01555f6415 2023-04-10 02:55:44,384 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/49dce2fb4fa54fac93ffa8478ed89520 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/49dce2fb4fa54fac93ffa8478ed89520 2023-04-10 02:55:44,384 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/4027af0d7287416fa564b2045c1be52b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/4027af0d7287416fa564b2045c1be52b 2023-04-10 02:55:44,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-10 02:55:44,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:44,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f91f1c15c884b2146b2b8d0ffac5d524, disabling compactions & flushes 2023-04-10 02:55:44,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:44,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:44,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. after waiting 0 ms 2023-04-10 02:55:44,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:44,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing f91f1c15c884b2146b2b8d0ffac5d524 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-10 02:55:44,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=A 2023-04-10 02:55:44,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:44,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=B 2023-04-10 02:55:44,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:44,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f91f1c15c884b2146b2b8d0ffac5d524, store=C 2023-04-10 02:55:44,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:44,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-10 02:55:44,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410b7d314b8b38c4c9dbf66645218d4bd9b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410b7d314b8b38c4c9dbf66645218d4bd9b 2023-04-10 02:55:44,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-10 02:55:45,274 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=482, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/019ed12d239f4b81916c1ec1d18bafa9 2023-04-10 02:55:45,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-10 02:55:45,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/b78902b1e9ab489dbe50947de481ec3b 2023-04-10 02:55:46,094 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/ec84256096934aedaaf13dd8adc57d80 2023-04-10 02:55:46,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/A/019ed12d239f4b81916c1ec1d18bafa9 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/019ed12d239f4b81916c1ec1d18bafa9 2023-04-10 02:55:46,101 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/019ed12d239f4b81916c1ec1d18bafa9, entries=100, sequenceid=482, filesize=18.6 K 2023-04-10 02:55:46,101 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/B/b78902b1e9ab489dbe50947de481ec3b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/b78902b1e9ab489dbe50947de481ec3b 2023-04-10 02:55:46,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/b78902b1e9ab489dbe50947de481ec3b, entries=100, sequenceid=482, filesize=9.5 K 2023-04-10 02:55:46,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/.tmp/C/ec84256096934aedaaf13dd8adc57d80 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/ec84256096934aedaaf13dd8adc57d80 2023-04-10 02:55:46,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/ec84256096934aedaaf13dd8adc57d80, entries=100, sequenceid=482, filesize=9.5 K 2023-04-10 02:55:46,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for f91f1c15c884b2146b2b8d0ffac5d524 in 1651ms, sequenceid=482, compaction requested=true 2023-04-10 02:55:46,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/recovered.edits/485.seqid, newMaxSeqId=485, maxSeqId=4 2023-04-10 02:55:46,116 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524. 2023-04-10 02:55:46,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f91f1c15c884b2146b2b8d0ffac5d524: 2023-04-10 02:55:46,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:46,118 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=f91f1c15c884b2146b2b8d0ffac5d524, regionState=CLOSED 2023-04-10 02:55:46,118 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095346118"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095346118"}]},"ts":"1681095346118"} 2023-04-10 02:55:46,120 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-10 02:55:46,120 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure f91f1c15c884b2146b2b8d0ffac5d524, server=jenkins-hbase4.apache.org,38321,1681095222501 in 1.8210 sec 2023-04-10 02:55:46,121 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-10 02:55:46,121 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f91f1c15c884b2146b2b8d0ffac5d524, UNASSIGN in 1.8310 sec 2023-04-10 02:55:46,122 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095346122"}]},"ts":"1681095346122"} 2023-04-10 02:55:46,123 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-10 02:55:46,125 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-10 02:55:46,126 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8440 sec 2023-04-10 02:55:46,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-10 02:55:46,390 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-10 02:55:46,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-10 02:55:46,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:46,393 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:46,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-10 02:55:46,394 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:46,397 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:46,399 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/recovered.edits] 2023-04-10 02:55:46,401 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/019ed12d239f4b81916c1ec1d18bafa9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/019ed12d239f4b81916c1ec1d18bafa9 2023-04-10 02:55:46,402 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/1b06dd5d959e490dbf586dfde6fbe3db to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/1b06dd5d959e490dbf586dfde6fbe3db 2023-04-10 02:55:46,403 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/581583e827554713bfc4019866768593 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/A/581583e827554713bfc4019866768593 2023-04-10 02:55:46,405 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/6243f04f856c455da2d8c5fd89506d3f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/6243f04f856c455da2d8c5fd89506d3f 2023-04-10 02:55:46,406 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/7fbfb2c3b65c4089883bde70e5be8822 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/7fbfb2c3b65c4089883bde70e5be8822 2023-04-10 02:55:46,407 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/b78902b1e9ab489dbe50947de481ec3b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/B/b78902b1e9ab489dbe50947de481ec3b 2023-04-10 02:55:46,409 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/b2fc7ee8d395475fbe00295a9a42b69e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/b2fc7ee8d395475fbe00295a9a42b69e 2023-04-10 02:55:46,410 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/b9179769db654664af4d012e8f76c3c9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/b9179769db654664af4d012e8f76c3c9 2023-04-10 02:55:46,411 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/ec84256096934aedaaf13dd8adc57d80 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/C/ec84256096934aedaaf13dd8adc57d80 2023-04-10 02:55:46,413 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/recovered.edits/485.seqid to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524/recovered.edits/485.seqid 2023-04-10 02:55:46,414 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/f91f1c15c884b2146b2b8d0ffac5d524 2023-04-10 02:55:46,414 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-10 02:55:46,414 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-10 02:55:46,415 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-10 02:55:46,417 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304100abdef583aa24d63bf8531c5d13a5a48 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304100abdef583aa24d63bf8531c5d13a5a48 2023-04-10 02:55:46,418 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410143202de8fc04978a8c97f405aa86a9c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410143202de8fc04978a8c97f405aa86a9c 2023-04-10 02:55:46,418 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304102e55d5dc0ae54f20b4cde16574de7124 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304102e55d5dc0ae54f20b4cde16574de7124 2023-04-10 02:55:46,419 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304103937613beebb4fbfb4148cd3ca6e198f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304103937613beebb4fbfb4148cd3ca6e198f 2023-04-10 02:55:46,420 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304104303bc5f5f60412d88039ce48bb7a5db to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304104303bc5f5f60412d88039ce48bb7a5db 2023-04-10 02:55:46,421 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410585619c2b93f47a8ad1302e6de5ad203 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410585619c2b93f47a8ad1302e6de5ad203 2023-04-10 02:55:46,422 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304106615c3a4feca438490b92512b73faf25 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304106615c3a4feca438490b92512b73faf25 2023-04-10 02:55:46,423 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304106d10c20f306c44ac8b6ab5998c7ba215 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304106d10c20f306c44ac8b6ab5998c7ba215 2023-04-10 02:55:46,424 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041088ec34eee4064610948089205c231d2e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041088ec34eee4064610948089205c231d2e 2023-04-10 02:55:46,424 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410904ac9c13c894b4eac59749377d10f48 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410904ac9c13c894b4eac59749377d10f48 2023-04-10 02:55:46,425 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304109c39d6fdc777433d917015ee8d1841eb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304109c39d6fdc777433d917015ee8d1841eb 2023-04-10 02:55:46,426 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a070f30322d84ce3a1e299f8c9e12790 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a070f30322d84ce3a1e299f8c9e12790 2023-04-10 02:55:46,427 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a0b842a5b40f4483981e54568d678ab3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a0b842a5b40f4483981e54568d678ab3 2023-04-10 02:55:46,428 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410b7d314b8b38c4c9dbf66645218d4bd9b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410b7d314b8b38c4c9dbf66645218d4bd9b 2023-04-10 02:55:46,429 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410cef6bf34b1ca4942a5fbed8f78d69a71 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410cef6bf34b1ca4942a5fbed8f78d69a71 2023-04-10 02:55:46,430 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410eacf2cf2538d4b07b5f7245ff47422b9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410eacf2cf2538d4b07b5f7245ff47422b9 2023-04-10 02:55:46,431 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fb0eb47e8cc749d8bebc09b2630b9a92 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fb0eb47e8cc749d8bebc09b2630b9a92 2023-04-10 02:55:46,432 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fd6986cc3c794cff98080962dde7237b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fd6986cc3c794cff98080962dde7237b 2023-04-10 02:55:46,433 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fe984059ccf9430dac8c705694945685 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fe984059ccf9430dac8c705694945685 2023-04-10 02:55:46,433 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-10 02:55:46,435 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:46,436 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-10 02:55:46,437 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-10 02:55:46,438 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:46,438 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-10 02:55:46,438 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681095346438"}]},"ts":"9223372036854775807"} 2023-04-10 02:55:46,440 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-10 02:55:46,440 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f91f1c15c884b2146b2b8d0ffac5d524, NAME => 'TestAcidGuarantees,,1681095319877.f91f1c15c884b2146b2b8d0ffac5d524.', STARTKEY => '', ENDKEY => ''}] 2023-04-10 02:55:46,440 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-10 02:55:46,440 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681095346440"}]},"ts":"9223372036854775807"} 2023-04-10 02:55:46,441 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-10 02:55:46,443 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:46,444 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 52 msec 2023-04-10 02:55:46,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-10 02:55:46,494 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-10 02:55:46,506 INFO [Listener at localhost/44533] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=236 (was 227) - Thread LEAK? -, OpenFileDescriptor=553 (was 558), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=417 (was 388) - SystemLoadAverage LEAK? -, ProcessCount=184 (was 184), AvailableMemoryMB=1319 (was 1373) 2023-04-10 02:55:46,518 INFO [Listener at localhost/44533] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=235, OpenFileDescriptor=552, MaxFileDescriptor=60000, SystemLoadAverage=417, ProcessCount=184, AvailableMemoryMB=1319 2023-04-10 02:55:46,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-10 02:55:46,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-10 02:55:46,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-10 02:55:46,522 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-10 02:55:46,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-10 02:55:46,522 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-10 02:55:46,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-10 02:55:46,523 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:46,524 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f empty. 2023-04-10 02:55:46,524 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:46,524 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-10 02:55:46,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-10 02:55:46,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-10 02:55:46,931 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-10 02:55:46,932 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 31f062e6d6eacf38c98ea51ae134e46f, NAME => 'TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp 2023-04-10 02:55:47,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-10 02:55:47,355 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:55:47,355 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 31f062e6d6eacf38c98ea51ae134e46f, disabling compactions & flushes 2023-04-10 02:55:47,355 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:47,355 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:47,355 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. after waiting 0 ms 2023-04-10 02:55:47,355 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:47,355 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:47,355 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:47,357 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-10 02:55:47,358 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095347358"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095347358"}]},"ts":"1681095347358"} 2023-04-10 02:55:47,359 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-10 02:55:47,359 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-10 02:55:47,359 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095347359"}]},"ts":"1681095347359"} 2023-04-10 02:55:47,360 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-10 02:55:47,363 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=31f062e6d6eacf38c98ea51ae134e46f, ASSIGN}] 2023-04-10 02:55:47,365 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=31f062e6d6eacf38c98ea51ae134e46f, ASSIGN 2023-04-10 02:55:47,366 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=31f062e6d6eacf38c98ea51ae134e46f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38321,1681095222501; forceNewPlan=false, retain=false 2023-04-10 02:55:47,517 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=31f062e6d6eacf38c98ea51ae134e46f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:47,517 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095347517"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095347517"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095347517"}]},"ts":"1681095347517"} 2023-04-10 02:55:47,518 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:55:47,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-10 02:55:47,672 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:47,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 31f062e6d6eacf38c98ea51ae134e46f, NAME => 'TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.', STARTKEY => '', ENDKEY => ''} 2023-04-10 02:55:47,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:47,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:55:47,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:47,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:47,673 INFO [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:47,674 DEBUG [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A 2023-04-10 02:55:47,674 DEBUG [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A 2023-04-10 02:55:47,675 INFO [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:55:47,675 INFO [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 31f062e6d6eacf38c98ea51ae134e46f columnFamilyName A 2023-04-10 02:55:47,675 INFO [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] regionserver.HStore(310): Store=31f062e6d6eacf38c98ea51ae134e46f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:55:47,675 INFO [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:47,676 DEBUG [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B 2023-04-10 02:55:47,676 DEBUG [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B 2023-04-10 02:55:47,677 INFO [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:55:47,677 INFO [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 31f062e6d6eacf38c98ea51ae134e46f columnFamilyName B 2023-04-10 02:55:47,677 INFO [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] regionserver.HStore(310): Store=31f062e6d6eacf38c98ea51ae134e46f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:55:47,677 INFO [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:47,678 DEBUG [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C 2023-04-10 02:55:47,678 DEBUG [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C 2023-04-10 02:55:47,678 INFO [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:55:47,678 INFO [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 31f062e6d6eacf38c98ea51ae134e46f columnFamilyName C 2023-04-10 02:55:47,679 INFO [StoreOpener-31f062e6d6eacf38c98ea51ae134e46f-1] regionserver.HStore(310): Store=31f062e6d6eacf38c98ea51ae134e46f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:55:47,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:47,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:47,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:47,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-10 02:55:47,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:47,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-10 02:55:47,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 31f062e6d6eacf38c98ea51ae134e46f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64781779, jitterRate=-0.03467626869678497}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-10 02:55:47,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:47,685 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., pid=49, masterSystemTime=1681095347669 2023-04-10 02:55:47,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:47,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:47,687 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=31f062e6d6eacf38c98ea51ae134e46f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:47,687 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095347687"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681095347687"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681095347687"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681095347687"}]},"ts":"1681095347687"} 2023-04-10 02:55:47,689 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-10 02:55:47,689 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 in 170 msec 2023-04-10 02:55:47,690 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-10 02:55:47,690 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=31f062e6d6eacf38c98ea51ae134e46f, ASSIGN in 326 msec 2023-04-10 02:55:47,690 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-10 02:55:47,690 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095347690"}]},"ts":"1681095347690"} 2023-04-10 02:55:47,691 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-10 02:55:47,693 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-10 02:55:47,694 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1730 sec 2023-04-10 02:55:48,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-10 02:55:48,627 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-10 02:55:48,628 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x34ad4aa8 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:48,632 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52203cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:48,633 DEBUG [Listener at localhost/44533] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:48,635 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56194, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:48,636 DEBUG [Listener at localhost/44533] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-10 02:55:48,638 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57650, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-10 02:55:48,639 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x1c29757d to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:48,643 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420f945a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:48,643 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x06edd017 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:48,647 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51b8df2a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:48,647 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x0fb168ce to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:48,650 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67346c47, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:48,651 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x461f5f3f to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:48,654 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@426cb82a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:48,654 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x63093e21 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:48,659 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f5363d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:48,659 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x211582f2 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:48,662 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@659c68ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:48,662 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x65816b6e to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:48,666 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@680e773e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:48,666 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x653bb8fc to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:48,670 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b8f652e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:48,670 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x720d29f5 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:48,675 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ac98310, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:48,675 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x3011deb3 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:55:48,681 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b480bc5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:55:48,685 DEBUG [hconnection-0x4cf1c817-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:48,685 DEBUG [hconnection-0x1aef6b75-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:48,686 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56200, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:48,686 DEBUG [hconnection-0x26009ef1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:48,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:48,687 DEBUG [hconnection-0x11f8fe3e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:48,687 DEBUG [hconnection-0x23bb825f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:48,687 DEBUG [hconnection-0x33d11841-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:48,688 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56232, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:48,688 DEBUG [hconnection-0x4df9337f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:48,688 DEBUG [hconnection-0xd7fea4c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:48,688 DEBUG [hconnection-0x5be0823c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:48,689 DEBUG [hconnection-0x3af3e5da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:55:48,689 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56256, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:48,695 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56216, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:48,695 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56198, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:48,695 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56238, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:48,696 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56224, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:48,696 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56258, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:48,696 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56244, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:48,696 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56260, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:55:48,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:48,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:48,701 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:48,701 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:48,701 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:48,701 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:48,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:48,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:48,706 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:48,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:48,706 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:48,707 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:48,707 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:48,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:48,707 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:48,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:48,707 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:48,707 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:48,707 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:48,708 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:48,708 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:48,708 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:48,708 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:48,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:48,709 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:48,709 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:55:48,710 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:48,710 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:48,710 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:48,710 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:48,710 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:48,710 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:48,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095408727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095408728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095408728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095408728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095408729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095348720, max timestamp=1681095348728Num uniques 859; 2023-04-10 02:55:48,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:55:48,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095348720, max timestamp=1681095348728Num uniques 859; 2023-04-10 02:55:48,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095348720, max timestamp=1681095348728Num uniques 859; 2023-04-10 02:55:48,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:55:48,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:55:48,732 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/1f80bca6608440b7a4a086832c18e4eb 2023-04-10 02:55:48,760 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/d49dfb784e2745599090498385cbc685 2023-04-10 02:55:48,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095408837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095408838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095408839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095408839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095408840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095408945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095408945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095408946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095408946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:48,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:48,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095408946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095409150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095409150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095409151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095409151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095409152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,178 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/4efa26ea214d4ce3b75a87ea146f355d 2023-04-10 02:55:49,182 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/1f80bca6608440b7a4a086832c18e4eb as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/1f80bca6608440b7a4a086832c18e4eb 2023-04-10 02:55:49,185 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/1f80bca6608440b7a4a086832c18e4eb, entries=150, sequenceid=12, filesize=11.6 K 2023-04-10 02:55:49,185 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/d49dfb784e2745599090498385cbc685 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d49dfb784e2745599090498385cbc685 2023-04-10 02:55:49,189 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d49dfb784e2745599090498385cbc685, entries=150, sequenceid=12, filesize=11.6 K 2023-04-10 02:55:49,189 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/4efa26ea214d4ce3b75a87ea146f355d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/4efa26ea214d4ce3b75a87ea146f355d 2023-04-10 02:55:49,192 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/4efa26ea214d4ce3b75a87ea146f355d, entries=150, sequenceid=12, filesize=11.6 K 2023-04-10 02:55:49,193 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 31f062e6d6eacf38c98ea51ae134e46f in 484ms, sequenceid=12, compaction requested=false 2023-04-10 02:55:49,193 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:49,193 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:49,195 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,195 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:49,195 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,195 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@717fbb56[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:49,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:49,196 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,198 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:49,198 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:49,198 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:49,198 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,199 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:49,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:49,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:49,200 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:49,202 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:49,202 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:49,202 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:49,202 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:49,202 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:49,204 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,204 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:49,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,204 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:49,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:49,205 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,206 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,207 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:49,207 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:49,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:49,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:49,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:49,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,208 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:49,208 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:49,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:49,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:49,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,208 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:49,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:49,308 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:49,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:49,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:49,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:49,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:49,311 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:49,311 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:49,311 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:49,311 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:49,312 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,312 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:49,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,314 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:49,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:49,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,315 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:49,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:49,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:49,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:49,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:49,316 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:49,316 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:49,316 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:49,317 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:49,317 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:49,317 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:49,317 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:49,317 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-10 02:55:49,317 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:49,317 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:49,317 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:49,317 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:49,318 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:49,318 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:55:49,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:49,457 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:49,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095409463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095409464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095409465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095409470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095409470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095409571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095409571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095409571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095409571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095409571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,725 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/fd4734768ed64813b7c4ac1d7613a276 2023-04-10 02:55:49,735 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/2a544e8b22284de3a642144c42e34a3d 2023-04-10 02:55:49,745 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/517aca225b7c40c99ba6bf44d25262ba 2023-04-10 02:55:49,748 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/fd4734768ed64813b7c4ac1d7613a276 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/fd4734768ed64813b7c4ac1d7613a276 2023-04-10 02:55:49,751 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/fd4734768ed64813b7c4ac1d7613a276, entries=150, sequenceid=46, filesize=11.6 K 2023-04-10 02:55:49,752 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/2a544e8b22284de3a642144c42e34a3d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2a544e8b22284de3a642144c42e34a3d 2023-04-10 02:55:49,756 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2a544e8b22284de3a642144c42e34a3d, entries=150, sequenceid=46, filesize=11.6 K 2023-04-10 02:55:49,757 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/517aca225b7c40c99ba6bf44d25262ba as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/517aca225b7c40c99ba6bf44d25262ba 2023-04-10 02:55:49,759 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/517aca225b7c40c99ba6bf44d25262ba, entries=150, sequenceid=46, filesize=11.6 K 2023-04-10 02:55:49,760 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=60.38 KB/61830 for 31f062e6d6eacf38c98ea51ae134e46f in 443ms, sequenceid=46, compaction requested=false 2023-04-10 02:55:49,760 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:49,760 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:49,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:49,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:49,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:49,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:49,762 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,762 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:49,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:49,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e8d701[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:49,764 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:49,764 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,765 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:49,766 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,766 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:49,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:49,779 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-10 02:55:49,779 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:49,779 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:49,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,779 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:49,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:49,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:49,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:49,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:49,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:49,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:49,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:49,783 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:49,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:49,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:49,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:49,783 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:49,785 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,785 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:49,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,785 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:49,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:49,786 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:49,787 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/ba0255fef08e4f1f9cf008204b70ad51 2023-04-10 02:55:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:49,789 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,792 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:49,792 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:49,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:49,793 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:49,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:49,793 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:49,793 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:49,793 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:49,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:49,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:49,793 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:49,793 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:49,793 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,793 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,793 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,793 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095409798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095409798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095409800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095409805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095409806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,893 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:49,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:49,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:49,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:49,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:49,897 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:49,897 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:49,897 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:49,897 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:49,897 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,897 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:49,898 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,898 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:49,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:49,899 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,899 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:49,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:49,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:49,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:49,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:49,900 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:49,900 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:49,900 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:49,900 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:49,903 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3be4f076[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:49,904 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:49,904 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,906 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,906 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:49,906 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:49,906 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:49,906 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,906 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:49,908 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:49,911 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:49,911 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:49,911 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:49,911 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:49,911 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:49,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095409907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095409907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,913 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095409908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,913 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:49,913 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:49,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:49,914 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,915 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:49,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095409910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,918 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,918 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:49,918 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:49,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:49,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:49,918 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:49,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:49,918 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:49,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:49,918 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:49,918 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:49,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095409913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,918 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:49,918 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:49,918 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:49,918 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:49,919 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,018 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:50,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:50,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:50,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:50,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:50,022 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:50,022 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:50,022 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:50,022 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:50,022 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,022 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,025 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,025 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:50,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:50,025 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,025 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:50,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:50,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:50,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:50,026 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,026 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:50,027 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:50,027 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:50,029 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:50,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37164939[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:50,031 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:50,031 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,032 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:50,033 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,033 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:50,035 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:50,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:50,037 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:50,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:50,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:50,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:50,037 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:50,039 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,039 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,040 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:50,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:50,040 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,042 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,045 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:50,045 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:50,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:50,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:50,045 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:50,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:50,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:50,046 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,046 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:50,046 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,046 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,046 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,046 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:50,046 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,046 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:50,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:50,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095410113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095410113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095410114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095410117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095410119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,146 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:50,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:50,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:50,150 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:50,150 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:50,150 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:50,150 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:50,150 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,150 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,152 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:50,152 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:50,152 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,152 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:50,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:50,153 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,154 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:50,154 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:50,154 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:50,156 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:50,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bbff186[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:50,159 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:50,159 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,160 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:50,160 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,161 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:50,162 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,162 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:50,162 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,162 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:50,164 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:50,165 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:50,165 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:50,165 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:50,165 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:50,169 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,169 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,169 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,170 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,171 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:50,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:50,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:50,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:50,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:50,173 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:50,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:50,174 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:50,174 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:50,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:50,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/894738f7084a40079a270a31fabece46 2023-04-10 02:55:50,274 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:50,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:50,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:50,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:50,277 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:50,277 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:50,277 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:50,277 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:50,278 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,278 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,279 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:50,279 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:50,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:50,279 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,279 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:50,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:50,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:50,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:50,281 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,281 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:50,281 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:50,281 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,282 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64b0692f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:50,284 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:50,284 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,286 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:50,286 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,286 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:50,287 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:50,288 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:50,290 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:50,290 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:50,290 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:50,290 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:50,290 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:50,291 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,291 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,292 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:50,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,292 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,294 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,296 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,296 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:50,296 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:50,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:50,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:50,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:50,296 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:50,296 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,296 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,296 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:50,296 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:50,296 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:50,296 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,297 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,297 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,397 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:50,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:50,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:50,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:50,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:50,400 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:50,400 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:50,400 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:50,400 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:50,400 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,400 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,403 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:50,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:50,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:50,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:50,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:50,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:50,404 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,404 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:50,405 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:50,405 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:50,406 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,407 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:50,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@784b528f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:50,408 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:50,408 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:50,409 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,410 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:50,411 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:50,411 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:50,411 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,411 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:50,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:50,414 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:50,414 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:50,414 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:50,414 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:50,414 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:50,416 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,416 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,416 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:50,416 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095410417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,421 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:50,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:50,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,423 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:50,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:50,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:50,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:50,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,423 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:50,423 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:50,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:50,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095410421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095410421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095410422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095410423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,523 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:50,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:50,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:50,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:50,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:50,526 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:50,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:50,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:50,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:50,527 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,527 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:50,528 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:50,528 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,528 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:50,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:50,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:50,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:50,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:50,530 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,530 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:50,530 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:50,530 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:50,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:50,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:50,530 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:50,531 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,531 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,532 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,532 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:50,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d11833e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:50,533 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:50,533 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,535 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,535 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:50,535 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:50,535 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:50,535 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,535 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:50,536 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,536 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:50,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:50,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:50,538 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:50,538 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:50,538 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:50,539 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:50,539 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:50,540 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,540 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:50,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,540 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:50,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:50,541 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,542 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:50,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:50,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:50,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:50,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:50,547 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:50,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,547 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:50,547 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:50,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,548 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:50,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:50,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:50,548 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:50,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/b1fcaaf359eb419cae28e15e6b44afb5 2023-04-10 02:55:50,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/ba0255fef08e4f1f9cf008204b70ad51 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ba0255fef08e4f1f9cf008204b70ad51 2023-04-10 02:55:50,624 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ba0255fef08e4f1f9cf008204b70ad51, entries=300, sequenceid=61, filesize=18.5 K 2023-04-10 02:55:50,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/894738f7084a40079a270a31fabece46 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/894738f7084a40079a270a31fabece46 2023-04-10 02:55:50,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/894738f7084a40079a270a31fabece46, entries=150, sequenceid=61, filesize=11.6 K 2023-04-10 02:55:50,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/b1fcaaf359eb419cae28e15e6b44afb5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/b1fcaaf359eb419cae28e15e6b44afb5 2023-04-10 02:55:50,631 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/b1fcaaf359eb419cae28e15e6b44afb5, entries=150, sequenceid=61, filesize=11.6 K 2023-04-10 02:55:50,632 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 31f062e6d6eacf38c98ea51ae134e46f in 853ms, sequenceid=61, compaction requested=true 2023-04-10 02:55:50,632 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:50,632 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:50,632 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:50,632 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:50,632 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:50,632 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:50,633 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:50,633 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:50,633 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/A is initiating minor compaction (all files) 2023-04-10 02:55:50,633 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/B is initiating minor compaction (all files) 2023-04-10 02:55:50,633 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/A in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,633 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/B in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,633 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/1f80bca6608440b7a4a086832c18e4eb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/fd4734768ed64813b7c4ac1d7613a276, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ba0255fef08e4f1f9cf008204b70ad51] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=41.7 K 2023-04-10 02:55:50,633 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d49dfb784e2745599090498385cbc685, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2a544e8b22284de3a642144c42e34a3d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/894738f7084a40079a270a31fabece46] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=34.8 K 2023-04-10 02:55:50,634 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 1f80bca6608440b7a4a086832c18e4eb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681095348702 2023-04-10 02:55:50,634 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d49dfb784e2745599090498385cbc685, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681095348702 2023-04-10 02:55:50,634 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting fd4734768ed64813b7c4ac1d7613a276, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681095348837 2023-04-10 02:55:50,634 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 2a544e8b22284de3a642144c42e34a3d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681095348837 2023-04-10 02:55:50,634 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ba0255fef08e4f1f9cf008204b70ad51, keycount=300, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681095349462 2023-04-10 02:55:50,634 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 894738f7084a40079a270a31fabece46, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681095349777 2023-04-10 02:55:50,640 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#A#compaction#392 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:50,641 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#B#compaction#393 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:50,648 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:50,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:50,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:50,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:50,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:50,652 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:50,652 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:50,652 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:50,652 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:50,652 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,652 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,654 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:50,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:50,654 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,654 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,655 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:50,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:50,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:50,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:50,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:50,655 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:50,655 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:50,655 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:50,656 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:50,656 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:50,656 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,656 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:50,656 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-10 02:55:50,656 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/6513ea61fb7b49799d43c9a50294e731 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/6513ea61fb7b49799d43c9a50294e731 2023-04-10 02:55:50,656 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:50,657 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:50,658 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:50,658 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:50,658 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:50,658 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:50,660 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/B of 31f062e6d6eacf38c98ea51ae134e46f into 6513ea61fb7b49799d43c9a50294e731(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:50,660 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:50,660 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/B, priority=13, startTime=1681095350632; duration=0sec 2023-04-10 02:55:50,660 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:50,660 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:50,661 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:50,661 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/C is initiating minor compaction (all files) 2023-04-10 02:55:50,661 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/C in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:50,661 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/4efa26ea214d4ce3b75a87ea146f355d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/517aca225b7c40c99ba6bf44d25262ba, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/b1fcaaf359eb419cae28e15e6b44afb5] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=34.8 K 2023-04-10 02:55:50,661 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 4efa26ea214d4ce3b75a87ea146f355d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681095348702 2023-04-10 02:55:50,662 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 517aca225b7c40c99ba6bf44d25262ba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681095348837 2023-04-10 02:55:50,664 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting b1fcaaf359eb419cae28e15e6b44afb5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681095349777 2023-04-10 02:55:50,666 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/861a51d7eefa475b9d81747d4f35b64d 2023-04-10 02:55:50,669 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#C#compaction#395 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:50,677 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/fbf7977a10a247bd8dc9c19a05d70fa6 2023-04-10 02:55:50,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:50,924 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:50,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095410936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095410937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095410943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095410943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:50,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:50,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095410945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095411046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095411046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095411048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,053 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/b956828c407f46cfaab8b10d38f711f1 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/b956828c407f46cfaab8b10d38f711f1 2023-04-10 02:55:51,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095411050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,057 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/A of 31f062e6d6eacf38c98ea51ae134e46f into b956828c407f46cfaab8b10d38f711f1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:51,057 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:51,057 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/A, priority=13, startTime=1681095350632; duration=0sec 2023-04-10 02:55:51,057 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:51,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095411056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,076 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/633b95d5d0194217b324e0c497f60059 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/633b95d5d0194217b324e0c497f60059 2023-04-10 02:55:51,080 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/C of 31f062e6d6eacf38c98ea51ae134e46f into 633b95d5d0194217b324e0c497f60059(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:51,080 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:51,080 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/C, priority=13, startTime=1681095350632; duration=0sec 2023-04-10 02:55:51,080 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:51,085 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/0633939e6bd94b19ad4d29a34b4b4764 2023-04-10 02:55:51,087 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/861a51d7eefa475b9d81747d4f35b64d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/861a51d7eefa475b9d81747d4f35b64d 2023-04-10 02:55:51,090 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/861a51d7eefa475b9d81747d4f35b64d, entries=150, sequenceid=78, filesize=11.6 K 2023-04-10 02:55:51,090 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/fbf7977a10a247bd8dc9c19a05d70fa6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/fbf7977a10a247bd8dc9c19a05d70fa6 2023-04-10 02:55:51,093 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/fbf7977a10a247bd8dc9c19a05d70fa6, entries=150, sequenceid=78, filesize=11.6 K 2023-04-10 02:55:51,094 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/0633939e6bd94b19ad4d29a34b4b4764 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/0633939e6bd94b19ad4d29a34b4b4764 2023-04-10 02:55:51,097 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/0633939e6bd94b19ad4d29a34b4b4764, entries=150, sequenceid=78, filesize=11.6 K 2023-04-10 02:55:51,097 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 31f062e6d6eacf38c98ea51ae134e46f in 441ms, sequenceid=78, compaction requested=false 2023-04-10 02:55:51,097 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:51,097 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:51,098 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:51,098 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:51,098 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,098 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:51,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:51,101 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,101 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:51,102 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:51,102 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39667edb[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:51,103 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:51,103 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:51,104 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:51,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:51,105 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,105 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:51,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:51,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:51,108 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:51,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:51,111 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:51,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:51,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:51,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:51,111 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:51,112 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:51,112 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:51,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:51,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:51,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:51,112 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:51,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:51,113 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:51,114 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:51,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:51,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:51,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:51,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:51,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:51,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:51,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:51,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:51,124 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:51,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:51,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:51,125 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:51,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:51,125 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:51,225 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:51,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:51,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:51,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:51,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:51,228 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:51,228 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:51,228 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:51,228 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:51,228 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:51,228 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:51,230 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,230 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:51,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:51,230 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,230 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:51,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:51,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:51,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:51,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:51,231 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:51,231 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:51,231 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:51,231 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:51,231 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:51,231 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:51,231 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:51,231 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-10 02:55:51,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:51,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:51,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:51,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:51,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:51,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:51,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:51,251 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:51,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095411266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095411267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095411268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095411268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095411275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095411376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095411376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095411376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095411376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095411384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095411580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095411580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095411581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095411581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095411588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,641 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/fd3d9bd2555d4ff298c2109ee75efb98 2023-04-10 02:55:51,650 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-10 02:55:51,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095411886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095411886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095411888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095411888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:51,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:51,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095411893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,055 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/bd730a5648d34241963bb01b2b3d0d31 2023-04-10 02:55:52,066 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/2f52354cb7444ca08e493572b555a856 2023-04-10 02:55:52,069 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/fd3d9bd2555d4ff298c2109ee75efb98 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/fd3d9bd2555d4ff298c2109ee75efb98 2023-04-10 02:55:52,072 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/fd3d9bd2555d4ff298c2109ee75efb98, entries=150, sequenceid=96, filesize=11.6 K 2023-04-10 02:55:52,073 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/bd730a5648d34241963bb01b2b3d0d31 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/bd730a5648d34241963bb01b2b3d0d31 2023-04-10 02:55:52,076 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/bd730a5648d34241963bb01b2b3d0d31, entries=150, sequenceid=96, filesize=11.6 K 2023-04-10 02:55:52,078 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/2f52354cb7444ca08e493572b555a856 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/2f52354cb7444ca08e493572b555a856 2023-04-10 02:55:52,082 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/2f52354cb7444ca08e493572b555a856, entries=150, sequenceid=96, filesize=11.6 K 2023-04-10 02:55:52,083 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 31f062e6d6eacf38c98ea51ae134e46f in 852ms, sequenceid=96, compaction requested=true 2023-04-10 02:55:52,083 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:52,083 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:52,086 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:52,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:52,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ef891c9[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:52,090 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:52,090 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,091 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:52,091 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,091 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:52,099 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,099 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:52,099 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:52,099 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:52,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:52,101 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:52,101 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:52,101 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:52,101 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:52,101 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:52,103 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,103 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:52,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,103 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:52,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:52,103 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:52,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,106 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:52,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:52,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:52,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:52,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:52,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:52,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,107 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,107 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:52,107 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:52,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:52,207 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:52,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:52,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:52,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:52,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:52,210 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:52,210 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:52,210 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:52,210 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:52,211 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,211 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,212 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:52,212 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:52,212 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,213 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:52,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:52,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:52,214 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:52,214 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:52,214 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:52,214 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:52,214 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:52,214 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:52,214 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:52,214 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-10 02:55:52,214 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:52,214 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:52,215 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:52,215 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:52,215 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:52,215 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:52,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:52,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:52,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095412406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095412406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095412406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095412409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095412413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095412513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095412513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095412514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095412514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095412517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,624 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/dff2d463a1ef40a28ed2b37cf5088a11 2023-04-10 02:55:52,639 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/195f341bd07d4db3b9d210e73f28e5b8 2023-04-10 02:55:52,661 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/d351f3b016e7493484712b06121a3bb3 2023-04-10 02:55:52,665 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/dff2d463a1ef40a28ed2b37cf5088a11 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/dff2d463a1ef40a28ed2b37cf5088a11 2023-04-10 02:55:52,668 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/dff2d463a1ef40a28ed2b37cf5088a11, entries=150, sequenceid=113, filesize=11.6 K 2023-04-10 02:55:52,668 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/195f341bd07d4db3b9d210e73f28e5b8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/195f341bd07d4db3b9d210e73f28e5b8 2023-04-10 02:55:52,672 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/195f341bd07d4db3b9d210e73f28e5b8, entries=150, sequenceid=113, filesize=11.6 K 2023-04-10 02:55:52,674 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/d351f3b016e7493484712b06121a3bb3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/d351f3b016e7493484712b06121a3bb3 2023-04-10 02:55:52,678 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/d351f3b016e7493484712b06121a3bb3, entries=150, sequenceid=113, filesize=11.6 K 2023-04-10 02:55:52,679 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 31f062e6d6eacf38c98ea51ae134e46f in 465ms, sequenceid=113, compaction requested=true 2023-04-10 02:55:52,679 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:52,679 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:52,679 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:52,679 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:52,679 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,679 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:52,679 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:52,681 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:52,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,681 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:52,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:52,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c056fb3[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:52,682 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:52,682 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,683 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:52,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:52,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:52,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:52,686 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,686 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:52,686 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:52,686 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:52,689 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:52,689 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:52,689 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:52,690 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:52,690 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:52,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:52,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:52,692 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,693 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,695 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:52,695 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:52,695 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:52,695 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:52,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:52,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:52,695 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,696 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:52,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:52,696 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:52,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:52,728 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-10 02:55:52,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:52,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:52,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:52,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:52,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:52,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:52,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095412743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095412750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095412750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095412751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095412752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,796 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:52,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:52,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:52,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:52,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:52,799 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:52,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:52,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:52,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:52,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,805 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,805 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:52,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:52,805 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,805 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:52,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:52,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:52,806 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:52,806 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:52,806 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:52,806 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:52,806 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:52,806 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:52,806 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:52,806 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:52,807 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:52,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:52,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:52,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:52,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:52,810 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:52,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:52,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dfc1b7e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:52,811 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:52,812 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,813 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,813 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:52,813 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:52,813 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,813 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:52,813 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:52,814 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:52,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:52,825 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:52,825 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:52,825 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:52,825 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:52,825 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:52,827 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:52,827 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,827 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:52,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:52,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:52,829 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,830 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:52,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,832 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:52,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:52,832 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:52,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:52,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:52,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:52,833 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:52,833 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:52,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095412856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095412859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095412859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095412859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:52,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095412859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,933 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:52,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:52,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:52,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:52,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:52,936 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:52,936 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:52,936 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:52,936 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:52,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,938 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:52,938 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:52,938 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,938 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:52,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:52,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:52,939 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:52,940 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:52,940 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:52,940 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:52,940 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:52,941 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:52,944 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:52,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:52,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:52,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7663cbc[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:52,951 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:52,951 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,953 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,953 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:52,954 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,954 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:52,954 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:52,954 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:52,955 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:52,955 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:52,955 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,955 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:52,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:52,957 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:52,958 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:52,958 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:52,958 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:52,958 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:52,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:52,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:52,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:52,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:52,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:52,960 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,961 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,963 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,963 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,963 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:52,963 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:52,963 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:52,963 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:52,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:52,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:52,963 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:52,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:52,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:52,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:52,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:52,964 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:52,964 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:52,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,064 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:53,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:53,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:53,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:53,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:53,068 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:53,068 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,068 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:53,068 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:53,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095413063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,068 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,068 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095413063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095413064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095413064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095413064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,071 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,071 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,071 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,071 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:53,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:53,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:53,072 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:53,072 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,072 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:53,073 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,073 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:53,073 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:53,073 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:53,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,074 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,075 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,075 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:53,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d8a8340[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:53,076 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:53,076 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,077 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,077 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:53,077 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:53,077 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,077 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:53,077 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:53,084 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:53,084 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,084 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,084 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:53,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:53,086 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:53,086 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:53,087 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:53,087 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:53,087 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:53,089 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,089 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,089 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,090 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,091 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,093 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,093 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,093 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,093 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,093 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,093 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,093 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:53,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:53,093 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,093 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,093 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,094 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:53,094 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:53,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:53,137 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/83cecfba865e496cb6a214b423730817 2023-04-10 02:55:53,194 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:53,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:53,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:53,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:53,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:53,197 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:53,197 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,197 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:53,197 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:53,197 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,197 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,198 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,199 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,199 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:53,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:53,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:53,200 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,200 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:53,200 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:53,200 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:53,201 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:53,211 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:53,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aa50fb5[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:53,212 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:53,212 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,213 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,213 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:53,214 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:53,214 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:53,214 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,214 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:53,215 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,215 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:53,215 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,215 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:53,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:53,217 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:53,217 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:53,217 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:53,217 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:53,217 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:53,220 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,220 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,220 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,221 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,222 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,224 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:53,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:53,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,225 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:53,225 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:53,325 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:53,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:53,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:53,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:53,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:53,328 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:53,328 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,328 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:53,328 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:53,328 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,328 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,330 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,330 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,330 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,330 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:53,331 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:53,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:53,331 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:53,331 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,331 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:53,332 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:53,332 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:53,332 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,332 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:53,332 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:53,332 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,332 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:53,332 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:53,332 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,332 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:53,332 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:53,334 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:53,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27efca74[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:53,336 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:53,336 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,338 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,338 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:53,338 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,339 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:53,338 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:53,339 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:53,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,340 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:53,340 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,340 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:53,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:53,344 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:53,344 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:53,344 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:53,344 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:53,344 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:53,346 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,346 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,346 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,346 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,348 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,350 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,350 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,351 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:53,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,351 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:53,351 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:53,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:53,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095413369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095413369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095413370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095413372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095413374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,451 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:53,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:53,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:53,455 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:53,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:53,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:53,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,458 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,458 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,458 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,458 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:53,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:53,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:53,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:53,459 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,460 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:53,460 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:53,460 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,460 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:53,460 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:53,460 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,460 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:53,460 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:53,461 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,462 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,462 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:53,462 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:53,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11da91e9[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:53,464 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:53,464 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,465 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:53,465 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,465 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:53,467 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:53,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:53,470 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:53,470 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:53,470 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:53,470 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:53,471 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:53,473 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,473 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,473 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,474 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,475 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,478 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,478 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,478 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,479 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,479 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,479 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,479 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,479 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,479 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,480 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:53,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:53,480 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:53,480 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:53,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/94b091a6a2d34ce8a979acd2ea8af2c9 2023-04-10 02:55:53,580 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:53,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:53,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:53,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:53,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:53,583 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:53,583 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,583 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:53,583 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:53,583 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,583 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,585 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,585 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,585 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,585 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:53,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:53,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:53,586 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,586 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:53,586 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:53,587 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,589 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:53,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58039011[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:53,590 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:53,590 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,591 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,592 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:53,592 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:53,592 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:53,592 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,592 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:53,600 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:53,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:53,607 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:53,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:53,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:53,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:53,607 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:53,609 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,609 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,609 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,610 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,611 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,616 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,616 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,616 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,616 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,616 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,616 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,616 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,616 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,616 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,617 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:53,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:53,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,617 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:53,617 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,717 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:53,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:53,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:53,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:53,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:53,720 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:53,720 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,720 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:53,720 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:53,720 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,721 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,722 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,722 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,722 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,722 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:53,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:53,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:53,723 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,723 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:53,723 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:53,723 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,724 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:53,724 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:53,725 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,725 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,726 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,726 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:53,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ab9f403[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:53,727 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:53,727 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,728 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:53,728 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,728 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:53,731 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:53,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:53,733 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:53,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:53,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:53,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:53,733 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:53,734 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,735 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,735 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,735 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,738 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:53,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:53,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:53,739 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:53,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,739 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:53,839 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:53,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:53,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:53,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:53,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:53,842 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:53,842 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,842 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:53,842 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:53,842 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,842 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,844 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,844 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,844 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,844 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:53,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:53,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:53,845 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,845 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:53,845 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:53,845 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:53,848 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,848 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,848 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,848 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:53,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@635c354a[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:53,850 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:53,850 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,851 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:53,851 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,851 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:53,852 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,852 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:53,852 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,852 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:53,854 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:53,855 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:53,855 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:53,855 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:53,855 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:53,856 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,856 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,856 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,856 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,857 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,860 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,861 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,861 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:53,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:53,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:53,861 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,861 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,861 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:53,861 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:53,861 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,861 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,861 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,861 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,861 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095413874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095413877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095413878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095413878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:53,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095413880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,955 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/d7200dd177b74b569ea24465488e56fe 2023-04-10 02:55:53,959 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/83cecfba865e496cb6a214b423730817 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/83cecfba865e496cb6a214b423730817 2023-04-10 02:55:53,961 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:53,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:53,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/83cecfba865e496cb6a214b423730817, entries=200, sequenceid=129, filesize=13.9 K 2023-04-10 02:55:53,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/94b091a6a2d34ce8a979acd2ea8af2c9 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/94b091a6a2d34ce8a979acd2ea8af2c9 2023-04-10 02:55:53,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:53,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:53,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:53,964 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:53,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:53,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:53,965 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,965 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,966 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,966 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/94b091a6a2d34ce8a979acd2ea8af2c9, entries=150, sequenceid=129, filesize=11.7 K 2023-04-10 02:55:53,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,966 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,966 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,967 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/d7200dd177b74b569ea24465488e56fe as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/d7200dd177b74b569ea24465488e56fe 2023-04-10 02:55:53,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:53,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:53,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:53,968 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,968 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:53,968 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:53,968 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,969 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:53,969 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:53,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/d7200dd177b74b569ea24465488e56fe, entries=150, sequenceid=129, filesize=11.7 K 2023-04-10 02:55:53,970 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,970 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,970 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,970 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,970 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 31f062e6d6eacf38c98ea51ae134e46f in 1242ms, sequenceid=129, compaction requested=true 2023-04-10 02:55:53,971 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:53,971 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,971 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:53,971 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:55:53,971 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:55:53,971 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:53,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@264ca511[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:53,972 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:53,972 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,972 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59583 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:55:53,972 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/B is initiating minor compaction (all files) 2023-04-10 02:55:53,972 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/B in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,973 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61923 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:55:53,973 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/6513ea61fb7b49799d43c9a50294e731, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/fbf7977a10a247bd8dc9c19a05d70fa6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/bd730a5648d34241963bb01b2b3d0d31, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/195f341bd07d4db3b9d210e73f28e5b8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/94b091a6a2d34ce8a979acd2ea8af2c9] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=58.2 K 2023-04-10 02:55:53,973 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/A is initiating minor compaction (all files) 2023-04-10 02:55:53,973 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/A in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:53,973 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/b956828c407f46cfaab8b10d38f711f1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/861a51d7eefa475b9d81747d4f35b64d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/fd3d9bd2555d4ff298c2109ee75efb98, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/dff2d463a1ef40a28ed2b37cf5088a11, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/83cecfba865e496cb6a214b423730817] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=60.5 K 2023-04-10 02:55:53,973 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6513ea61fb7b49799d43c9a50294e731, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681095349777 2023-04-10 02:55:53,973 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,973 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting b956828c407f46cfaab8b10d38f711f1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681095349777 2023-04-10 02:55:53,973 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:53,974 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting fbf7977a10a247bd8dc9c19a05d70fa6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681095349788 2023-04-10 02:55:53,974 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,974 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:53,974 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 861a51d7eefa475b9d81747d4f35b64d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681095349788 2023-04-10 02:55:53,974 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting bd730a5648d34241963bb01b2b3d0d31, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681095350934 2023-04-10 02:55:53,974 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting fd3d9bd2555d4ff298c2109ee75efb98, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681095350934 2023-04-10 02:55:53,974 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 195f341bd07d4db3b9d210e73f28e5b8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681095351267 2023-04-10 02:55:53,975 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting dff2d463a1ef40a28ed2b37cf5088a11, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681095351267 2023-04-10 02:55:53,975 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 94b091a6a2d34ce8a979acd2ea8af2c9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1681095352408 2023-04-10 02:55:53,975 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 83cecfba865e496cb6a214b423730817, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1681095352405 2023-04-10 02:55:53,976 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:53,976 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,976 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:53,977 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:53,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:53,979 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:53,979 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:53,979 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:53,979 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:53,979 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:53,983 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,983 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,983 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:53,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,984 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,986 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,987 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#A#compaction#407 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:53,987 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#B#compaction#408 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:53,988 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,988 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,988 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:53,988 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:53,988 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:53,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:53,988 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:53,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:53,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:53,988 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:53,988 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:53,988 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:53,988 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:53,988 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:53,989 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:53,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,001 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/ceccbe5c8c8244eab0b494e8a88639bd as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/ceccbe5c8c8244eab0b494e8a88639bd 2023-04-10 02:55:54,006 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/B of 31f062e6d6eacf38c98ea51ae134e46f into ceccbe5c8c8244eab0b494e8a88639bd(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:54,006 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:54,006 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/B, priority=11, startTime=1681095353971; duration=0sec 2023-04-10 02:55:54,006 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:54,007 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:55:54,009 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/ba32a4b60c8c4d9389d3f11f05ebb3fd as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ba32a4b60c8c4d9389d3f11f05ebb3fd 2023-04-10 02:55:54,009 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59583 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:55:54,009 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/C is initiating minor compaction (all files) 2023-04-10 02:55:54,010 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/C in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,010 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/633b95d5d0194217b324e0c497f60059, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/0633939e6bd94b19ad4d29a34b4b4764, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/2f52354cb7444ca08e493572b555a856, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/d351f3b016e7493484712b06121a3bb3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/d7200dd177b74b569ea24465488e56fe] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=58.2 K 2023-04-10 02:55:54,010 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 633b95d5d0194217b324e0c497f60059, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681095349777 2023-04-10 02:55:54,010 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 0633939e6bd94b19ad4d29a34b4b4764, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681095349788 2023-04-10 02:55:54,011 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 2f52354cb7444ca08e493572b555a856, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681095350934 2023-04-10 02:55:54,012 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d351f3b016e7493484712b06121a3bb3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681095351267 2023-04-10 02:55:54,012 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d7200dd177b74b569ea24465488e56fe, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1681095352408 2023-04-10 02:55:54,014 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/A of 31f062e6d6eacf38c98ea51ae134e46f into ba32a4b60c8c4d9389d3f11f05ebb3fd(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:54,014 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:54,014 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/A, priority=11, startTime=1681095353971; duration=0sec 2023-04-10 02:55:54,014 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:54,031 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#C#compaction#409 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:54,043 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/f705569c77ea4b708d261351201a4a68 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f705569c77ea4b708d261351201a4a68 2023-04-10 02:55:54,047 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/C of 31f062e6d6eacf38c98ea51ae134e46f into f705569c77ea4b708d261351201a4a68(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:54,047 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:54,047 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/C, priority=11, startTime=1681095353971; duration=0sec 2023-04-10 02:55:54,047 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:54,089 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:54,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:54,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:54,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:54,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:54,092 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:54,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:54,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:54,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,093 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,093 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,094 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,094 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:54,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:54,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:54,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:54,095 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,095 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:54,095 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-10 02:55:54,096 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:54,096 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:54,096 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:54,096 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:54,096 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:54,096 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:54,103 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/331abe4fd04244cda36c014ad7f478ac 2023-04-10 02:55:54,113 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/1dd19299ae0846fbaec3559bf8523f7a 2023-04-10 02:55:54,132 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/f0d57df5829842ed96047c2c0487691c 2023-04-10 02:55:54,139 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/331abe4fd04244cda36c014ad7f478ac as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/331abe4fd04244cda36c014ad7f478ac 2023-04-10 02:55:54,142 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/331abe4fd04244cda36c014ad7f478ac, entries=150, sequenceid=148, filesize=11.8 K 2023-04-10 02:55:54,142 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/1dd19299ae0846fbaec3559bf8523f7a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/1dd19299ae0846fbaec3559bf8523f7a 2023-04-10 02:55:54,146 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/1dd19299ae0846fbaec3559bf8523f7a, entries=150, sequenceid=148, filesize=11.8 K 2023-04-10 02:55:54,149 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/f0d57df5829842ed96047c2c0487691c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f0d57df5829842ed96047c2c0487691c 2023-04-10 02:55:54,151 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f0d57df5829842ed96047c2c0487691c, entries=150, sequenceid=148, filesize=11.8 K 2023-04-10 02:55:54,152 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 31f062e6d6eacf38c98ea51ae134e46f in 57ms, sequenceid=148, compaction requested=false 2023-04-10 02:55:54,152 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:54,152 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:54,156 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,157 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,157 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:54,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@620b2ad1[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:54,158 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:54,158 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,160 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:54,162 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,163 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:54,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:54,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:54,163 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:54,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:54,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:54,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:54,163 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:54,164 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,164 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,164 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,165 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,166 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,170 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,170 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,170 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,170 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:54,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:54,170 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,170 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,170 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,170 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,170 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:54,171 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:54,170 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,171 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:54,271 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:54,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:54,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:54,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:54,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:54,274 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:54,274 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,274 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:54,274 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:54,275 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,275 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,276 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,276 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,277 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,277 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:54,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:54,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:54,278 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:54,278 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,278 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:54,279 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,279 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:54,279 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:54,279 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:54,281 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,281 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,281 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,281 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1767d37f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:54,284 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:54,284 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,286 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,286 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:54,286 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:54,286 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:54,286 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,286 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:54,287 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:54,287 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,287 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,287 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:54,291 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:54,291 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:54,291 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:54,291 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:54,291 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:54,293 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,293 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,293 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,294 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,295 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,298 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,298 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,298 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:54,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:54,298 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,298 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,298 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:54,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,299 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:54,299 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:54,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,399 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:54,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:54,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:54,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:54,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:54,408 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:54,408 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,408 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:54,408 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:54,408 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,408 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,410 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,410 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,410 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,410 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:54,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:54,412 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,412 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:54,412 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:54,412 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:54,414 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:54,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@159342bc[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:54,416 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:54,416 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,417 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:54,417 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:54,417 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:54,419 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,419 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:54,420 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,420 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:54,420 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:54,420 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:54,420 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:54,420 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:54,420 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:54,420 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:54,420 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:54,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,424 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,430 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:54,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:54,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,430 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:54,430 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:54,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:54,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,530 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:54,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:54,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:54,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:54,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:54,534 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:54,534 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,534 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:54,534 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:54,534 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,534 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,535 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,535 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:54,537 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,537 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:54,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:54,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:54,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:54,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:54,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:54,541 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,541 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,541 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,541 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:54,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56340932[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:54,543 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:54,543 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,545 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,545 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,545 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:54,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,546 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:54,546 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:54,546 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:54,547 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,547 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:54,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:54,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:54,551 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:54,551 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:54,551 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:54,551 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:54,551 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:54,554 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,554 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,554 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,554 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,556 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,558 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,558 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,558 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,558 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,558 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,558 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:54,558 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,558 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:54,559 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,559 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:54,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,559 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:54,559 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:54,659 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:54,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:54,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:54,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:54,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:54,664 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:54,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:54,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:54,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,667 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,667 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,667 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,667 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:54,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:54,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:54,669 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:54,669 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,669 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:54,669 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:54,669 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,669 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:54,669 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:54,669 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,669 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:54,669 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:54,670 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,671 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,671 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:54,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d52ec87[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:54,672 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:54,672 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,673 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,673 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,673 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:54,673 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:54,675 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:54,678 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:54,678 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:54,678 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:54,678 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:54,678 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:54,680 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,680 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,680 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,680 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,682 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,684 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,684 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,684 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,684 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,684 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:54,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:54,684 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,684 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:54,684 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,684 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:54,685 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,685 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:54,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,685 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,785 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:54,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:54,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:54,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:54,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:54,788 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:54,788 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,788 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:54,788 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:54,789 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,789 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,790 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,791 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:54,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:54,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:54,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:54,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:54,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:54,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:54,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:54,792 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,792 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:54,792 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:54,792 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,792 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:54,792 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:54,792 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,792 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:54,792 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:54,793 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,793 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44872679[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:54,795 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:54,795 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,797 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:54,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,797 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:54,799 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:54,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:54,805 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:54,805 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:54,805 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:54,805 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:54,805 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:54,807 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,807 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,807 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,807 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,809 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,812 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,812 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,812 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,812 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:54,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:54,812 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,812 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,812 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,812 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,812 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:54,812 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,812 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:54,812 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:54,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,912 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:54,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:54,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:54,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:54,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:54,915 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:54,915 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,915 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:54,915 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:54,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:54,915 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:55:54,916 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,916 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,916 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:54,916 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:54,916 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:54,916 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:54,917 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:54,917 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:54,919 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,919 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:54,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:54,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:54,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:54,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:54,922 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,922 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:54,922 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:54,922 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:54,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:54,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:54,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:54,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,923 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:54,924 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,924 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,924 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,924 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,926 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/ae4c5c3715bf4326b999602cb19fd1b1 2023-04-10 02:55:54,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:54,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b8e1af7[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:54,926 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:54,926 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,927 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:54,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:54,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:54,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,929 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,929 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:54,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:54,934 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:54,934 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:54,934 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:54,934 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:54,934 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:54,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:54,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:54,940 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:54,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:54,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:54,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:54,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,942 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:54,943 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:54,943 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:54,943 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:54,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:54,943 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:54,943 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:54,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:54,944 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/c57a79a9e9f64796a124ef4be968a5aa 2023-04-10 02:55:54,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:54,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095414944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:54,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095414945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:54,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095414946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:54,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095414947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:54,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095414948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:54,957 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/a0056bc2463d43af8b42e31088fd7efe 2023-04-10 02:55:54,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/ae4c5c3715bf4326b999602cb19fd1b1 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ae4c5c3715bf4326b999602cb19fd1b1 2023-04-10 02:55:54,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ae4c5c3715bf4326b999602cb19fd1b1, entries=200, sequenceid=166, filesize=14.1 K 2023-04-10 02:55:54,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/c57a79a9e9f64796a124ef4be968a5aa as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/c57a79a9e9f64796a124ef4be968a5aa 2023-04-10 02:55:54,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/c57a79a9e9f64796a124ef4be968a5aa, entries=150, sequenceid=166, filesize=11.8 K 2023-04-10 02:55:54,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/a0056bc2463d43af8b42e31088fd7efe as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/a0056bc2463d43af8b42e31088fd7efe 2023-04-10 02:55:54,974 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/a0056bc2463d43af8b42e31088fd7efe, entries=150, sequenceid=166, filesize=11.8 K 2023-04-10 02:55:54,975 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 31f062e6d6eacf38c98ea51ae134e46f in 60ms, sequenceid=166, compaction requested=true 2023-04-10 02:55:54,975 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:54,975 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:54,975 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:54,975 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:54,975 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:54,975 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:54,977 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38672 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:54,977 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/A is initiating minor compaction (all files) 2023-04-10 02:55:54,978 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36282 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:54,978 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/A in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,978 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/B is initiating minor compaction (all files) 2023-04-10 02:55:54,978 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ba32a4b60c8c4d9389d3f11f05ebb3fd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/331abe4fd04244cda36c014ad7f478ac, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ae4c5c3715bf4326b999602cb19fd1b1] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=37.8 K 2023-04-10 02:55:54,978 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/B in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:54,978 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/ceccbe5c8c8244eab0b494e8a88639bd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/1dd19299ae0846fbaec3559bf8523f7a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/c57a79a9e9f64796a124ef4be968a5aa] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=35.4 K 2023-04-10 02:55:54,978 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ba32a4b60c8c4d9389d3f11f05ebb3fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1681095352408 2023-04-10 02:55:54,978 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting ceccbe5c8c8244eab0b494e8a88639bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1681095352408 2023-04-10 02:55:54,978 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 331abe4fd04244cda36c014ad7f478ac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681095352742 2023-04-10 02:55:54,978 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 1dd19299ae0846fbaec3559bf8523f7a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681095352742 2023-04-10 02:55:54,979 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ae4c5c3715bf4326b999602cb19fd1b1, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681095354889 2023-04-10 02:55:54,979 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting c57a79a9e9f64796a124ef4be968a5aa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681095354890 2023-04-10 02:55:54,984 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#A#compaction#416 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:54,985 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#B#compaction#417 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:54,999 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/cc9abc9abbc64fb08650ee1de48bb2b0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/cc9abc9abbc64fb08650ee1de48bb2b0 2023-04-10 02:55:55,003 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/B of 31f062e6d6eacf38c98ea51ae134e46f into cc9abc9abbc64fb08650ee1de48bb2b0(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:55,003 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:55,003 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/B, priority=13, startTime=1681095354975; duration=0sec 2023-04-10 02:55:55,003 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:55,003 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:55,004 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36282 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:55,004 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/C is initiating minor compaction (all files) 2023-04-10 02:55:55,004 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/C in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,004 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f705569c77ea4b708d261351201a4a68, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f0d57df5829842ed96047c2c0487691c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/a0056bc2463d43af8b42e31088fd7efe] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=35.4 K 2023-04-10 02:55:55,004 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting f705569c77ea4b708d261351201a4a68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1681095352408 2023-04-10 02:55:55,005 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting f0d57df5829842ed96047c2c0487691c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681095352742 2023-04-10 02:55:55,005 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting a0056bc2463d43af8b42e31088fd7efe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681095354890 2023-04-10 02:55:55,010 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#C#compaction#418 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:55,018 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/076be910bc274927b143b022d268e758 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/076be910bc274927b143b022d268e758 2023-04-10 02:55:55,021 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/C of 31f062e6d6eacf38c98ea51ae134e46f into 076be910bc274927b143b022d268e758(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:55,021 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:55,021 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/C, priority=13, startTime=1681095354975; duration=0sec 2023-04-10 02:55:55,021 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:55,043 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:55,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:55,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:55,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:55,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:55,046 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:55,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:55,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:55,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:55,047 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,047 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,049 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:55,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,050 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,050 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:55,050 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:55,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:55,051 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,051 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:55,051 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-10 02:55:55,051 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:55,051 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:55,051 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:55,051 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:55,051 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:55,051 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:55,061 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/4091c4885357409db57927372ee54901 2023-04-10 02:55:55,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:55,063 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:55,072 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/d600ef5ee6664763807f307d2e26284c 2023-04-10 02:55:55,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095415065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095415070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095415071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095415072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095415073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,080 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/a10944042575431ca39379f68e6252c2 2023-04-10 02:55:55,083 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/4091c4885357409db57927372ee54901 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/4091c4885357409db57927372ee54901 2023-04-10 02:55:55,086 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/4091c4885357409db57927372ee54901, entries=150, sequenceid=188, filesize=11.8 K 2023-04-10 02:55:55,086 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/d600ef5ee6664763807f307d2e26284c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d600ef5ee6664763807f307d2e26284c 2023-04-10 02:55:55,089 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d600ef5ee6664763807f307d2e26284c, entries=150, sequenceid=188, filesize=11.8 K 2023-04-10 02:55:55,090 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/a10944042575431ca39379f68e6252c2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/a10944042575431ca39379f68e6252c2 2023-04-10 02:55:55,092 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/a10944042575431ca39379f68e6252c2, entries=150, sequenceid=188, filesize=11.8 K 2023-04-10 02:55:55,093 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 31f062e6d6eacf38c98ea51ae134e46f in 42ms, sequenceid=188, compaction requested=false 2023-04-10 02:55:55,093 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:55,093 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:55,095 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,095 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,096 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,096 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d85f72d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:55,098 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:55,098 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,099 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,099 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:55,099 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:55,099 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:55,099 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,099 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:55,100 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:55,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,100 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:55,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:55,103 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:55,103 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:55,103 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:55,103 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:55,103 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:55,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:55,106 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,108 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,109 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:55,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:55,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:55,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:55,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:55,110 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:55,110 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:55,110 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:55,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:55,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:55,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:55,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-10 02:55:55,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:55,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:55,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:55,178 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:55,178 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:55,178 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:55,195 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/8aea0de88c9a4c81b6b133afaa1b07b2 2023-04-10 02:55:55,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/9fd7e9be720748f2b41bb65f3ab68ca9 2023-04-10 02:55:55,210 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:55,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:55,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:55,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:55,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:55,216 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:55,216 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:55,216 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:55,216 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:55,216 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,217 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,218 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:55,218 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,218 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,218 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:55,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:55,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095415211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095415211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095415212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095415212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095415213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:55,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:55,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:55,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:55,234 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,234 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,234 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,234 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:55,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47c912ce[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:55,236 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:55,236 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,238 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:55,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:55,238 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:55,239 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,239 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:55,240 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:55,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:55,242 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:55,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:55,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:55,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:55,242 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:55,244 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,244 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,244 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:55,245 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,246 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,248 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,248 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,248 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,248 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:55,248 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:55,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:55,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:55,248 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:55,249 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:55,249 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:55,248 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:55,248 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,249 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,249 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:55,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095415321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095415321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095415322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095415323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095415323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,349 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:55,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:55,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:55,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:55,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:55,352 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:55,352 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:55,352 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:55,352 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:55,352 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,352 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,355 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:55,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:55,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:55,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:55,357 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,357 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:55,357 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:55,357 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,357 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:55,357 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:55,357 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,357 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:55,357 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:55,364 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a8040a0[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:55,365 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:55,366 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:55,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:55,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:55,367 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,367 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:55,369 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,369 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:55,369 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,369 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:55,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:55,371 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:55,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:55,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:55,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:55,371 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:55,373 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,373 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,373 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,375 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,377 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,380 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:55,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:55,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:55,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:55,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:55,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:55,381 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:55,381 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:55,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:55,391 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/3c785b1a1e5246d08bbf62b8616704b3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/3c785b1a1e5246d08bbf62b8616704b3 2023-04-10 02:55:55,396 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/A of 31f062e6d6eacf38c98ea51ae134e46f into 3c785b1a1e5246d08bbf62b8616704b3(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:55,396 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:55,396 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/A, priority=13, startTime=1681095354975; duration=0sec 2023-04-10 02:55:55,396 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:55,481 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:55,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:55,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:55,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:55,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:55,484 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:55,484 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:55,484 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:55,484 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:55,485 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,485 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,486 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:55,486 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:55,486 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,487 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:55,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:55,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:55,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:55,488 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:55,488 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:55,488 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:55,492 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:55,492 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:55,492 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,492 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:55,492 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:55,492 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,492 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:55,492 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:55,492 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,492 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:55,492 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:55,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,494 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:55,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e2aca5[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:55,495 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:55,495 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,496 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:55,500 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:55,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:55,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:55,502 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:55,502 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:55,502 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:55,503 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:55,503 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:55,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:55,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,505 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,505 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,508 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,508 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,508 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:55,508 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,508 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:55,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:55,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:55,508 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,508 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,508 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,508 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:55,508 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:55,508 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:55,508 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:55,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:55,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095415528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095415528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095415528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095415528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095415529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,608 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:55,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:55,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:55,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:55,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:55,613 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:55,613 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:55,613 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:55,613 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:55,613 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,613 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,614 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:55,614 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:55,615 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,615 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:55,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:55,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:55,616 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,616 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:55,616 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:55,616 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:55,618 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,618 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:55,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13cc1f2b[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:55,620 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:55,620 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,622 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,622 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:55,622 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:55,622 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:55,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:55,623 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,623 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,623 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:55,623 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:55,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:55,626 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:55,626 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:55,626 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:55,626 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:55,626 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:55,627 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,627 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,628 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:55,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,628 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:55,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,630 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,633 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:55,633 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:55,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:55,633 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:55,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:55,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:55,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:55,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:55,634 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:55,634 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:55,634 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,634 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,634 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,635 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,635 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,635 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:55,635 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,635 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/5e3e7ee22c5a4ba6b16d8ea05c3b9dd8 2023-04-10 02:55:55,638 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/8aea0de88c9a4c81b6b133afaa1b07b2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/8aea0de88c9a4c81b6b133afaa1b07b2 2023-04-10 02:55:55,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/8aea0de88c9a4c81b6b133afaa1b07b2, entries=200, sequenceid=201, filesize=14.1 K 2023-04-10 02:55:55,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/9fd7e9be720748f2b41bb65f3ab68ca9 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/9fd7e9be720748f2b41bb65f3ab68ca9 2023-04-10 02:55:55,646 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/9fd7e9be720748f2b41bb65f3ab68ca9, entries=150, sequenceid=201, filesize=11.8 K 2023-04-10 02:55:55,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/5e3e7ee22c5a4ba6b16d8ea05c3b9dd8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/5e3e7ee22c5a4ba6b16d8ea05c3b9dd8 2023-04-10 02:55:55,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/5e3e7ee22c5a4ba6b16d8ea05c3b9dd8, entries=150, sequenceid=201, filesize=11.8 K 2023-04-10 02:55:55,650 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 31f062e6d6eacf38c98ea51ae134e46f in 473ms, sequenceid=201, compaction requested=true 2023-04-10 02:55:55,650 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:55,650 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:55,650 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:55,651 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:55,650 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:55,656 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:55,657 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38874 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:55,657 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:55,657 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/A is initiating minor compaction (all files) 2023-04-10 02:55:55,657 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/B is initiating minor compaction (all files) 2023-04-10 02:55:55,657 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/A in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,657 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/B in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,657 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/3c785b1a1e5246d08bbf62b8616704b3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/4091c4885357409db57927372ee54901, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/8aea0de88c9a4c81b6b133afaa1b07b2] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=38.0 K 2023-04-10 02:55:55,657 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/cc9abc9abbc64fb08650ee1de48bb2b0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d600ef5ee6664763807f307d2e26284c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/9fd7e9be720748f2b41bb65f3ab68ca9] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=35.6 K 2023-04-10 02:55:55,657 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 3c785b1a1e5246d08bbf62b8616704b3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681095354890 2023-04-10 02:55:55,658 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting cc9abc9abbc64fb08650ee1de48bb2b0, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681095354890 2023-04-10 02:55:55,658 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 4091c4885357409db57927372ee54901, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1681095354925 2023-04-10 02:55:55,658 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d600ef5ee6664763807f307d2e26284c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1681095354925 2023-04-10 02:55:55,658 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 8aea0de88c9a4c81b6b133afaa1b07b2, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681095355064 2023-04-10 02:55:55,658 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 9fd7e9be720748f2b41bb65f3ab68ca9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681095355064 2023-04-10 02:55:55,665 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#A#compaction#425 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:55,665 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#B#compaction#426 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:55,674 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/0a9ba0b37d05411daa5d58a3a2b7021e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/0a9ba0b37d05411daa5d58a3a2b7021e 2023-04-10 02:55:55,678 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/A of 31f062e6d6eacf38c98ea51ae134e46f into 0a9ba0b37d05411daa5d58a3a2b7021e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:55,678 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:55,678 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/A, priority=13, startTime=1681095355650; duration=0sec 2023-04-10 02:55:55,678 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:55:55,678 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:55:55,679 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:55:55,679 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/C is initiating minor compaction (all files) 2023-04-10 02:55:55,679 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/C in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,679 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/076be910bc274927b143b022d268e758, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/a10944042575431ca39379f68e6252c2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/5e3e7ee22c5a4ba6b16d8ea05c3b9dd8] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=35.6 K 2023-04-10 02:55:55,680 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 076be910bc274927b143b022d268e758, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681095354890 2023-04-10 02:55:55,680 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting a10944042575431ca39379f68e6252c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1681095354925 2023-04-10 02:55:55,680 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 5e3e7ee22c5a4ba6b16d8ea05c3b9dd8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681095355064 2023-04-10 02:55:55,684 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#C#compaction#427 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:55:55,695 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/847acd23978f4776aa04f3e1b29a3c12 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/847acd23978f4776aa04f3e1b29a3c12 2023-04-10 02:55:55,698 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/C of 31f062e6d6eacf38c98ea51ae134e46f into 847acd23978f4776aa04f3e1b29a3c12(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:55,698 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:55,698 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/C, priority=13, startTime=1681095355656; duration=0sec 2023-04-10 02:55:55,699 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:55,734 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:55,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:55,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:55,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:55,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:55,737 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:55,737 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:55,737 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:55,737 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:55,737 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,737 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,738 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:55,738 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:55,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,739 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:55,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:55,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:55,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:55,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:55,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:55,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:55,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:55,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:55,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:55,740 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:55,740 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:55,740 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-10 02:55:55,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:55,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:55,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:55,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:55,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:55,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:55,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:55,840 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:55,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095415844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095415848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095415848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095415849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095415850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095415951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095415955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095415955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095415960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:55,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:55,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095415960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,085 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/f34f066e73224342ae90bb0d38a03ad6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/f34f066e73224342ae90bb0d38a03ad6 2023-04-10 02:55:56,089 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/B of 31f062e6d6eacf38c98ea51ae134e46f into f34f066e73224342ae90bb0d38a03ad6(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:55:56,089 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:56,089 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/B, priority=13, startTime=1681095355650; duration=0sec 2023-04-10 02:55:56,090 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:55:56,148 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/5eb7149ca1d84d29a4fc569f4153f275 2023-04-10 02:55:56,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095416158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095416159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095416160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095416168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095416168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,178 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/0f4e3a6dcacd4e28943f2e906b9887e4 2023-04-10 02:55:56,228 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/12199aa8920346b7b7585bd2da60c155 2023-04-10 02:55:56,232 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/5eb7149ca1d84d29a4fc569f4153f275 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/5eb7149ca1d84d29a4fc569f4153f275 2023-04-10 02:55:56,236 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/5eb7149ca1d84d29a4fc569f4153f275, entries=150, sequenceid=224, filesize=11.8 K 2023-04-10 02:55:56,237 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/0f4e3a6dcacd4e28943f2e906b9887e4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/0f4e3a6dcacd4e28943f2e906b9887e4 2023-04-10 02:55:56,244 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/0f4e3a6dcacd4e28943f2e906b9887e4, entries=150, sequenceid=224, filesize=11.8 K 2023-04-10 02:55:56,244 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/12199aa8920346b7b7585bd2da60c155 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/12199aa8920346b7b7585bd2da60c155 2023-04-10 02:55:56,258 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/12199aa8920346b7b7585bd2da60c155, entries=150, sequenceid=224, filesize=11.8 K 2023-04-10 02:55:56,258 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 31f062e6d6eacf38c98ea51ae134e46f in 518ms, sequenceid=224, compaction requested=false 2023-04-10 02:55:56,259 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:56,259 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:56,259 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:56,259 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:56,259 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,259 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:56,259 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:56,261 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:56,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:56,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:56,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c9c30fe[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:56,268 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:56,268 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,271 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,271 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:56,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,272 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:56,272 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:56,272 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:56,281 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:56,282 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:56,282 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:56,282 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:56,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:56,287 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:56,287 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:56,287 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:56,287 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:56,287 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:56,288 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:56,289 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,289 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:56,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:56,290 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,292 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,295 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:56,295 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:56,295 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,295 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:56,295 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:56,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:56,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:56,295 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:56,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:56,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:56,295 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,296 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,296 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,296 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:56,296 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:56,296 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:56,396 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:56,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:56,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:56,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:56,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:56,399 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:56,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:56,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:56,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:56,400 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,400 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:56,401 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:56,401 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:56,401 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,401 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:56,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:56,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:56,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:56,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:56,408 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:56,408 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:56,408 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:56,408 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:56,408 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:55:56,408 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:56,408 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:56,409 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:56,409 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:56,409 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:56,409 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:56,465 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:56,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:56,472 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/018fd21d6b8444a3b2438e266541eb7c 2023-04-10 02:55:56,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095416490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095416492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095416492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095416495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095416496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,505 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/e9d50c9cf9bc40c6898546030820628a 2023-04-10 02:55:56,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095416600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095416601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095416601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095416603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095416604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095416803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095416804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095416805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095416810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:56,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095416810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,926 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/afe2c25a759940aeae3541de719c0cb2 2023-04-10 02:55:56,930 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/018fd21d6b8444a3b2438e266541eb7c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/018fd21d6b8444a3b2438e266541eb7c 2023-04-10 02:55:56,933 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/018fd21d6b8444a3b2438e266541eb7c, entries=150, sequenceid=237, filesize=11.8 K 2023-04-10 02:55:56,933 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/e9d50c9cf9bc40c6898546030820628a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/e9d50c9cf9bc40c6898546030820628a 2023-04-10 02:55:56,936 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/e9d50c9cf9bc40c6898546030820628a, entries=150, sequenceid=237, filesize=11.8 K 2023-04-10 02:55:56,937 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/afe2c25a759940aeae3541de719c0cb2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/afe2c25a759940aeae3541de719c0cb2 2023-04-10 02:55:56,940 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/afe2c25a759940aeae3541de719c0cb2, entries=150, sequenceid=237, filesize=11.8 K 2023-04-10 02:55:56,940 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 31f062e6d6eacf38c98ea51ae134e46f in 532ms, sequenceid=237, compaction requested=true 2023-04-10 02:55:56,940 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:56,940 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:56,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:56,940 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:56,941 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,941 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:56,941 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:56,943 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,943 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:56,943 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,943 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:56,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:56,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:56,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:56,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:56,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:56,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24a1e68d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:56,945 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:56,945 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,946 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:56,946 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,946 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:56,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:56,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:56,948 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:56,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:56,950 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:56,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:56,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:56,950 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:56,950 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:56,952 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,952 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:56,952 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:56,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:56,952 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,958 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,958 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:56,958 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:56,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:56,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:56,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:56,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:56,958 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:56,958 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:56,958 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:56,958 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:56,958 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:56,959 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:56,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:56,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:56,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:56,958 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:56,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:57,059 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:57,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:57,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:57,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:57,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:57,062 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:57,062 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:57,062 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:57,062 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:57,062 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:57,062 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:57,064 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:57,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:57,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:57,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,064 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:57,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:57,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:57,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:57,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:57,065 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:57,065 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:57,065 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:57,065 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:57,065 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:57,065 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:57,065 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:57,065 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-10 02:55:57,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:57,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:57,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:57,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:57,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:57,066 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:57,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:57,112 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:57,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095417120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095417120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095417121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095417122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095417122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095417226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095417226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095417229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095417229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095417230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095417432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095417432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095417435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095417436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095417436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,476 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/6974b42651c64cbab0eb2b75b201ba86 2023-04-10 02:55:57,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095417738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095417739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095417741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095417741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:57,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095417742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:57,885 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/793d13a988844576a40398f6441e232d 2023-04-10 02:55:58,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:58,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095418242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:58,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56200 deadline: 1681095418244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:58,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095418250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:58,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56260 deadline: 1681095418250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:58,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56198 deadline: 1681095418250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,296 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/3d189fcc544940719fc1142446042568 2023-04-10 02:55:58,300 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/6974b42651c64cbab0eb2b75b201ba86 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/6974b42651c64cbab0eb2b75b201ba86 2023-04-10 02:55:58,303 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/6974b42651c64cbab0eb2b75b201ba86, entries=150, sequenceid=257, filesize=11.8 K 2023-04-10 02:55:58,303 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/793d13a988844576a40398f6441e232d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/793d13a988844576a40398f6441e232d 2023-04-10 02:55:58,307 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/793d13a988844576a40398f6441e232d, entries=150, sequenceid=257, filesize=11.8 K 2023-04-10 02:55:58,307 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/3d189fcc544940719fc1142446042568 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/3d189fcc544940719fc1142446042568 2023-04-10 02:55:58,310 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/3d189fcc544940719fc1142446042568, entries=150, sequenceid=257, filesize=11.8 K 2023-04-10 02:55:58,311 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 31f062e6d6eacf38c98ea51ae134e46f in 1246ms, sequenceid=257, compaction requested=true 2023-04-10 02:55:58,311 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:58,311 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:58,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:58,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:58,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:58,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:58,313 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:58,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30b2e138[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:58,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:58,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,315 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,315 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:58,316 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:58,316 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:58,316 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,316 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:58,319 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:58,321 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:58,322 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:58,322 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:58,322 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:58,322 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:58,323 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,323 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:58,323 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:58,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:58,324 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:58,325 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,327 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,327 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:58,327 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:58,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:58,327 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:58,327 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:58,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:58,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:58,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:58,327 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:58,327 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:58,328 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:58,327 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,328 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:58,328 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,328 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,428 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:58,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:58,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:58,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:58,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:58,431 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:58,431 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:58,431 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:58,431 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:58,431 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,431 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:58,432 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,432 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:58,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:58,433 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,433 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:58,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:58,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:58,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:58,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:58,434 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:58,434 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:58,434 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:58,434 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:58,434 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:58,434 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:58,434 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:58,434 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:55:58,434 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:58,434 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:58,434 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:58,434 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:58,435 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:58,435 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:58,440 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/9a1b04e14a6046f9a5960ed19421bcc5 2023-04-10 02:55:58,849 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/f8fa91016fd54112b02f2152afbd5f36 2023-04-10 02:55:58,862 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/bf1b565a73d0445d8271488286d23c6d 2023-04-10 02:55:58,866 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/9a1b04e14a6046f9a5960ed19421bcc5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/9a1b04e14a6046f9a5960ed19421bcc5 2023-04-10 02:55:58,869 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/9a1b04e14a6046f9a5960ed19421bcc5, entries=150, sequenceid=269, filesize=11.9 K 2023-04-10 02:55:58,870 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/f8fa91016fd54112b02f2152afbd5f36 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/f8fa91016fd54112b02f2152afbd5f36 2023-04-10 02:55:58,874 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/f8fa91016fd54112b02f2152afbd5f36, entries=150, sequenceid=269, filesize=11.9 K 2023-04-10 02:55:58,874 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/bf1b565a73d0445d8271488286d23c6d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/bf1b565a73d0445d8271488286d23c6d 2023-04-10 02:55:58,878 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/bf1b565a73d0445d8271488286d23c6d, entries=150, sequenceid=269, filesize=11.9 K 2023-04-10 02:55:58,878 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 31f062e6d6eacf38c98ea51ae134e46f in 444ms, sequenceid=269, compaction requested=true 2023-04-10 02:55:58,878 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:58,878 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:58,881 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,881 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:58,881 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,881 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:58,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:58,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43748542[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:58,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:58,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,885 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,885 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:58,885 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:58,885 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:58,886 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,886 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:58,887 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,887 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:58,887 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:58,887 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:58,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:58,890 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:58,890 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:58,890 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:58,890 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:58,890 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:58,895 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,895 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:58,895 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:58,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:58,896 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:58,897 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,900 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,900 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:58,900 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:58,900 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:58,900 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:58,900 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:58,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:58,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:58,900 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:58,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:58,900 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:58,900 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:58,900 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:58,900 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:58,900 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:58,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:59,000 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:59,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:59,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:59,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:59,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:59,004 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:59,004 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:59,004 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:59,004 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:59,004 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,004 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:59,005 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,005 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:59,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:59,006 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,006 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:59,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:59,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:59,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:59,007 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:59,007 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:59,007 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:59,007 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:59,007 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:59,007 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:59,007 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:59,007 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:59,007 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:59,008 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:59,008 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:59,008 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,008 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:59,008 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:59,009 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,009 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:59,009 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,010 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e125612[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:59,011 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:59,011 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,013 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:59,013 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,014 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:59,014 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:59,015 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,015 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:59,015 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:59,015 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:59,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:59,017 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:59,018 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:59,018 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:59,018 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:59,018 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:59,019 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:59,019 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,019 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:59,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:59,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:59,020 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:59,021 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:59,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,023 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,023 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:59,023 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:59,023 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,023 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,023 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:59,023 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:59,023 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:59,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:59,024 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:59,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:59,024 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:59,024 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:59,024 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:59,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:59,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:59,124 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:59,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:59,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:59,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:59,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:59,129 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:59,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:59,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:59,130 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:59,130 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,130 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:59,132 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,132 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:59,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:59,133 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,133 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:59,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:59,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:59,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:59,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:59,134 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:59,134 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:59,134 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:55:59,134 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:55:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:55:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:55:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:55:59,136 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,136 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:59,136 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,136 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:55:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15ac4ad3[Count = 0] remaining members to acquire global barrier 2023-04-10 02:55:59,137 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:55:59,138 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,139 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:55:59,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:55:59,139 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:55:59,139 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,139 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:55:59,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,141 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:55:59,141 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:59,141 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:59,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:59,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:59,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:59,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:55:59,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:55:59,143 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:55:59,143 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:55:59,143 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:55:59,143 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:55:59,143 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:55:59,151 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,151 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,151 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:55:59,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:59,151 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:55:59,152 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:55:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:55:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,155 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,155 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:59,155 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:59,155 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:59,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:59,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:55:59,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:55:59,155 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:59,155 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:55:59,155 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:55:59,155 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:55:59,155 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,155 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,156 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:55:59,156 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:55:59,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:55:59,256 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:55:59,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:55:59,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:55:59,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:55:59,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:55:59,259 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:55:59,259 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:59,259 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:55:59,259 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:55:59,259 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,260 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:59,261 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,261 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:55:59,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:55:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:55:59,261 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,261 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:55:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:59,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:55:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:55:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:55:59,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:55:59,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:55:59,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:55:59,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:55:59,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:55:59,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:55:59,267 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:55:59,267 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:55:59,267 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:55:59,267 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:55:59,267 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:55:59,267 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:55:59,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:55:59,268 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:55:59,268 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:59,268 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:55:59,268 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:59,268 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:55:59,268 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:55:59,291 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/06f4f84c19d64d2b9407a1ed0326d707 2023-04-10 02:55:59,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095359289, max timestamp=1681095359330Num uniques 859; 2023-04-10 02:55:59,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095359289, max timestamp=1681095359330Num uniques 859; 2023-04-10 02:55:59,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:59,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:59,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095359289, max timestamp=1681095359330Num uniques 859; 2023-04-10 02:55:59,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:55:59,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56216 deadline: 1681095419379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095419379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56256 deadline: 1681095419380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095419380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095419381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095419497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095419497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095419497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095419497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095419498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,706 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/7e32d3deddd04639b5c92b93b669709b 2023-04-10 02:55:59,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095419705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095419706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095419706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095419706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:55:59,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:55:59,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095419707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095420013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095420013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095420013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095420015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095420016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,115 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/db9941ec72b34bcfa5e6afcabdbffa82 2023-04-10 02:56:00,118 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/06f4f84c19d64d2b9407a1ed0326d707 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/06f4f84c19d64d2b9407a1ed0326d707 2023-04-10 02:56:00,121 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/06f4f84c19d64d2b9407a1ed0326d707, entries=250, sequenceid=281, filesize=16.7 K 2023-04-10 02:56:00,122 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/7e32d3deddd04639b5c92b93b669709b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7e32d3deddd04639b5c92b93b669709b 2023-04-10 02:56:00,125 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7e32d3deddd04639b5c92b93b669709b, entries=150, sequenceid=281, filesize=11.9 K 2023-04-10 02:56:00,126 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/db9941ec72b34bcfa5e6afcabdbffa82 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/db9941ec72b34bcfa5e6afcabdbffa82 2023-04-10 02:56:00,129 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/db9941ec72b34bcfa5e6afcabdbffa82, entries=150, sequenceid=281, filesize=11.9 K 2023-04-10 02:56:00,129 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 31f062e6d6eacf38c98ea51ae134e46f in 862ms, sequenceid=281, compaction requested=true 2023-04-10 02:56:00,129 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:00,129 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:00,129 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:00,130 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:00,130 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,130 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:00,130 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:00,131 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,131 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:00,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:00,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:00,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:00,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3230d27f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:00,133 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:00,133 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:00,134 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:00,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:00,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:00,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:00,134 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:00,135 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,135 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:00,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,137 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:00,137 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:00,137 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:00,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:00,139 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:00,139 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:00,139 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:00,139 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:00,139 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:00,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:00,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:00,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:00,141 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:00,141 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:00,142 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,147 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,147 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:00,147 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:00,147 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:00,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:00,147 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:00,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:00,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:00,147 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:00,147 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,147 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:00,147 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:00,147 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:00,147 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:00,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:00,147 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:00,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:00,247 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:00,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:00,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:00,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:00,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:00,250 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:00,250 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:00,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:00,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:00,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:00,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:00,253 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:00,253 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:00,253 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,253 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:00,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:00,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:00,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:00,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:00,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:00,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:00,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:00,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:00,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:00,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:00,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:00,255 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:00,255 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:00,255 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-10 02:56:00,255 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:00,255 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:00,255 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:00,255 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:00,255 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:00,255 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:00,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:00,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:00,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095420538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095420538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095420539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095420539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095420543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095420654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095420654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095420654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095420654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,664 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/8478bfeff2bb4931be02ed3d10752fd8 2023-04-10 02:56:00,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095420660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095420861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095420861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095420861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095420861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:00,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:00,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095420872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,083 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/99e8e740bdd041e09f14969ca41b4389 2023-04-10 02:56:01,102 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/537e26d840e948538b7ae7d47ad5e82d 2023-04-10 02:56:01,106 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/8478bfeff2bb4931be02ed3d10752fd8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/8478bfeff2bb4931be02ed3d10752fd8 2023-04-10 02:56:01,114 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/8478bfeff2bb4931be02ed3d10752fd8, entries=150, sequenceid=318, filesize=11.9 K 2023-04-10 02:56:01,114 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/99e8e740bdd041e09f14969ca41b4389 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/99e8e740bdd041e09f14969ca41b4389 2023-04-10 02:56:01,120 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/99e8e740bdd041e09f14969ca41b4389, entries=150, sequenceid=318, filesize=11.9 K 2023-04-10 02:56:01,121 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/537e26d840e948538b7ae7d47ad5e82d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/537e26d840e948538b7ae7d47ad5e82d 2023-04-10 02:56:01,124 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/537e26d840e948538b7ae7d47ad5e82d, entries=150, sequenceid=318, filesize=11.9 K 2023-04-10 02:56:01,124 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 31f062e6d6eacf38c98ea51ae134e46f in 869ms, sequenceid=318, compaction requested=true 2023-04-10 02:56:01,125 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:01,125 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:01,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:01,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:01,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:01,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:01,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,137 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,137 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,137 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:01,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55891fee[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:01,140 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:01,141 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,142 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,143 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:01,143 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:01,143 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:01,143 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,143 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:01,144 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,145 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:01,145 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,145 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:01,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:01,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:01,149 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:01,149 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:01,149 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:01,149 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:01,149 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:01,151 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,151 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,151 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,152 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,154 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,157 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,157 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:01,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:01,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:01,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:01,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:01,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:01,157 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,157 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:01,157 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:01,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,160 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,160 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,160 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,160 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,160 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,160 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,160 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:01,178 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:56:01,178 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:01,178 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:01,178 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:01,178 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:01,178 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:01,178 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:01,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681095361215, max timestamp=1681095361230Num uniques 859; 2023-04-10 02:56:01,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681095361215, max timestamp=1681095361230Num uniques 859; 2023-04-10 02:56:01,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-10 02:56:01,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-10 02:56:01,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681095361215, max timestamp=1681095361230Num uniques 859; 2023-04-10 02:56:01,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-10 02:56:01,258 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:01,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:01,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:01,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:01,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:01,261 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:01,261 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:01,261 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:01,261 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:01,261 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,261 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,264 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,264 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:01,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:01,264 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,264 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:01,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:01,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:01,266 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:01,266 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:01,266 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:01,268 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:01,268 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:01,268 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:01,268 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:01,268 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:01,268 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:01,268 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:01,268 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:01,268 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,268 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:01,268 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:01,270 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,270 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,271 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,271 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:01,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7987618b[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:01,272 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:01,272 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,274 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,274 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:01,274 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:01,274 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:01,275 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,275 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:01,276 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,276 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:01,276 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,276 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:01,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:01,279 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:01,279 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:01,279 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:01,279 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:01,279 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:01,283 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,283 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,283 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095421271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,285 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,296 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:01,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,299 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:01,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:01,299 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,300 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:01,300 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:01,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095421284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:01,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095421284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095421285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095421286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095421386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,400 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:01,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:01,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:01,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:01,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:01,403 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:01,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:01,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:01,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:01,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,404 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:01,404 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:01,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:01,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,405 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,405 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:01,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:01,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:01,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:01,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:01,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:01,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:01,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:01,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:01,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:01,407 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:01,407 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:01,407 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:01,407 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:01,407 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:01,407 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:01,407 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,407 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:01,407 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:01,408 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,409 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,409 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,409 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:01,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e0a46ba[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:01,410 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:01,410 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095421402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,412 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,413 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:01,413 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:01,413 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:01,413 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,413 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:01,414 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:01,414 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,414 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,414 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095421405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095421405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095421406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:01,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:01,417 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:01,417 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:01,417 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:01,417 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:01,417 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:01,418 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,418 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,418 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,419 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,420 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,429 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,429 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:01,429 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:01,429 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:01,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:01,429 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:01,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,430 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:01,430 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:01,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,430 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,530 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:01,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:01,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:01,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:01,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:01,533 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:01,533 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:01,533 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:01,533 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:01,534 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,534 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,535 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,535 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:01,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:01,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:01,535 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,535 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:01,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:01,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:01,537 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:01,537 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:01,537 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:01,537 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:01,539 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,539 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,539 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,539 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:01,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a1dce2e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:01,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:01,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,541 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:01,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:01,542 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:01,542 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,542 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:01,544 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:01,544 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,544 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,544 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:01,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:01,547 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:01,547 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:01,547 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:01,547 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:01,547 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:01,548 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,548 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,548 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,549 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,550 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,553 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:01,553 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:01,553 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,553 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,553 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:01,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:01,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:01,553 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,554 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,554 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,554 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,554 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,554 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:01,554 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:01,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/010e27a62b87462c8b16fcf6791ba52d 2023-04-10 02:56:01,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095421595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095421614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095421616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095421617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095421617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,654 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:01,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:01,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:01,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:01,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:01,657 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:01,657 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:01,658 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:01,658 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:01,659 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,659 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,660 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:01,660 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:01,660 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,660 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:01,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:01,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:01,662 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:01,662 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:01,662 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:01,662 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:01,665 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,665 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,665 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,665 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:01,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f6c0ec[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:01,667 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:01,667 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,668 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:01,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:01,668 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:01,668 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,668 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:01,669 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,669 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:01,670 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,670 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:01,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:01,672 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:01,672 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:01,672 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:01,672 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:01,672 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:01,674 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,674 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,674 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,675 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,676 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:01,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:01,679 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:01,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:01,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,680 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:01,680 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:01,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:01,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,780 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:01,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:01,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:01,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:01,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:01,783 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:01,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:01,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:01,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:01,783 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,784 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,785 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,785 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:01,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:01,785 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,785 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:01,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:01,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:01,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:01,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:01,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:01,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:01,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:01,786 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:01,786 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:01,787 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:01,787 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:01,787 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:01,787 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:01,787 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:01,787 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,787 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:01,787 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:01,788 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,789 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,789 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@653f76a0[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:01,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:01,790 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,791 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:01,791 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,791 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:01,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:01,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,794 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:01,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:01,796 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:01,796 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:01,796 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:01,796 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:01,796 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:01,798 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,798 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,798 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,798 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,799 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:01,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:01,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:01,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:01,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:01,801 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,802 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:01,802 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:01,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,802 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,802 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,902 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:01,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:01,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:01,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:01,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:01,905 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:01,905 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:01,905 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:01,905 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:01,905 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,905 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,907 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:01,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:01,907 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,907 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,907 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:01,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:01,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:01,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:01,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:01,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:01,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:01,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:01,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:01,908 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:01,908 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:01,908 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:01,908 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:01,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:01,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:01,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:01,908 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:01,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095421906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,915 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,915 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:01,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@395cd32b[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:01,917 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:01,917 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,920 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:01,920 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,920 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,922 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:01,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:01,924 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:01,924 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:01,924 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:01,924 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:01,924 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:01,928 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,928 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,928 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:01,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:01,929 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:01,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095421919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095421919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:01,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:01,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:01,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:01,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:01,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:01,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:01,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,931 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:01,931 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:01,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,932 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:01,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:01,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,932 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:01,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:01,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:01,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:01,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095421927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:01,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:01,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095421927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,025 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/36531af925c64dcb8aec0dd619fbfe87 2023-04-10 02:56:02,032 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:02,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:02,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:02,034 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/efc43354fba74453babea5c9a887b483 2023-04-10 02:56:02,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:02,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:02,035 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:02,035 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,035 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:02,035 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:02,036 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,036 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,037 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,037 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:02,038 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/010e27a62b87462c8b16fcf6791ba52d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/010e27a62b87462c8b16fcf6791ba52d 2023-04-10 02:56:02,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:02,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:02,038 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:02,038 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,038 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:02,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:02,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:02,039 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,039 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:02,039 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:02,039 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:02,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:02,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:02,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:02,040 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,040 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,041 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,041 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/010e27a62b87462c8b16fcf6791ba52d, entries=150, sequenceid=328, filesize=11.9 K 2023-04-10 02:56:02,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/36531af925c64dcb8aec0dd619fbfe87 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/36531af925c64dcb8aec0dd619fbfe87 2023-04-10 02:56:02,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:02,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d640690[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:02,043 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:02,043 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,045 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,045 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:02,045 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:02,045 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:02,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:02,046 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/36531af925c64dcb8aec0dd619fbfe87, entries=100, sequenceid=328, filesize=9.5 K 2023-04-10 02:56:02,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/efc43354fba74453babea5c9a887b483 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/efc43354fba74453babea5c9a887b483 2023-04-10 02:56:02,047 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:02,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,047 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:02,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:02,050 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:02,050 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:02,050 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:02,050 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:02,050 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:02,053 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/efc43354fba74453babea5c9a887b483, entries=100, sequenceid=328, filesize=9.5 K 2023-04-10 02:56:02,053 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,053 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,053 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,054 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for 31f062e6d6eacf38c98ea51ae134e46f in 876ms, sequenceid=328, compaction requested=true 2023-04-10 02:56:02,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,054 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,054 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,054 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-10 02:56:02,054 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,054 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-10 02:56:02,054 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:02,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,056 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,057 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 102242 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-10 02:56:02,057 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/A is initiating minor compaction (all files) 2023-04-10 02:56:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,057 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/A in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,057 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 94922 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-10 02:56:02,057 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/0a9ba0b37d05411daa5d58a3a2b7021e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/5eb7149ca1d84d29a4fc569f4153f275, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/018fd21d6b8444a3b2438e266541eb7c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/6974b42651c64cbab0eb2b75b201ba86, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/9a1b04e14a6046f9a5960ed19421bcc5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/06f4f84c19d64d2b9407a1ed0326d707, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/8478bfeff2bb4931be02ed3d10752fd8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/010e27a62b87462c8b16fcf6791ba52d] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=99.8 K 2023-04-10 02:56:02,057 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/B is initiating minor compaction (all files) 2023-04-10 02:56:02,057 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/B in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,057 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/f34f066e73224342ae90bb0d38a03ad6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/0f4e3a6dcacd4e28943f2e906b9887e4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/e9d50c9cf9bc40c6898546030820628a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/793d13a988844576a40398f6441e232d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/f8fa91016fd54112b02f2152afbd5f36, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7e32d3deddd04639b5c92b93b669709b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/99e8e740bdd041e09f14969ca41b4389, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/36531af925c64dcb8aec0dd619fbfe87] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=92.7 K 2023-04-10 02:56:02,057 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 0a9ba0b37d05411daa5d58a3a2b7021e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681095355064 2023-04-10 02:56:02,058 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting f34f066e73224342ae90bb0d38a03ad6, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681095355064 2023-04-10 02:56:02,058 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 5eb7149ca1d84d29a4fc569f4153f275, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1681095355188 2023-04-10 02:56:02,058 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 0f4e3a6dcacd4e28943f2e906b9887e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1681095355188 2023-04-10 02:56:02,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:02,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,059 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:02,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:02,060 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 018fd21d6b8444a3b2438e266541eb7c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1681095355840 2023-04-10 02:56:02,060 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting e9d50c9cf9bc40c6898546030820628a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1681095355840 2023-04-10 02:56:02,060 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:02,060 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:02,060 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 6974b42651c64cbab0eb2b75b201ba86, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681095356487 2023-04-10 02:56:02,060 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 793d13a988844576a40398f6441e232d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681095356487 2023-04-10 02:56:02,060 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 9a1b04e14a6046f9a5960ed19421bcc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1681095357119 2023-04-10 02:56:02,060 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting f8fa91016fd54112b02f2152afbd5f36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1681095357119 2023-04-10 02:56:02,061 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 06f4f84c19d64d2b9407a1ed0326d707, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1681095359264 2023-04-10 02:56:02,061 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 7e32d3deddd04639b5c92b93b669709b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1681095359265 2023-04-10 02:56:02,061 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 8478bfeff2bb4931be02ed3d10752fd8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681095359377 2023-04-10 02:56:02,061 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 99e8e740bdd041e09f14969ca41b4389, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681095359377 2023-04-10 02:56:02,061 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 010e27a62b87462c8b16fcf6791ba52d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1681095360538 2023-04-10 02:56:02,061 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 36531af925c64dcb8aec0dd619fbfe87, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1681095360542 2023-04-10 02:56:02,075 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#A#compaction#449 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:02,075 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#B#compaction#450 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:02,088 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/5645a46c898947f9aca252deed0dbbfe as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/5645a46c898947f9aca252deed0dbbfe 2023-04-10 02:56:02,092 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/B of 31f062e6d6eacf38c98ea51ae134e46f into 5645a46c898947f9aca252deed0dbbfe(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:02,092 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:02,092 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/B, priority=8, startTime=1681095362054; duration=0sec 2023-04-10 02:56:02,092 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:02,092 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-10 02:56:02,095 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 94922 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-10 02:56:02,095 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/C is initiating minor compaction (all files) 2023-04-10 02:56:02,095 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/C in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,095 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/847acd23978f4776aa04f3e1b29a3c12, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/12199aa8920346b7b7585bd2da60c155, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/afe2c25a759940aeae3541de719c0cb2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/3d189fcc544940719fc1142446042568, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/bf1b565a73d0445d8271488286d23c6d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/db9941ec72b34bcfa5e6afcabdbffa82, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/537e26d840e948538b7ae7d47ad5e82d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/efc43354fba74453babea5c9a887b483] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=92.7 K 2023-04-10 02:56:02,096 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 847acd23978f4776aa04f3e1b29a3c12, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681095355064 2023-04-10 02:56:02,096 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 12199aa8920346b7b7585bd2da60c155, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1681095355188 2023-04-10 02:56:02,096 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting afe2c25a759940aeae3541de719c0cb2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1681095355840 2023-04-10 02:56:02,096 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 3d189fcc544940719fc1142446042568, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681095356487 2023-04-10 02:56:02,097 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting bf1b565a73d0445d8271488286d23c6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1681095357119 2023-04-10 02:56:02,097 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting db9941ec72b34bcfa5e6afcabdbffa82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1681095359265 2023-04-10 02:56:02,097 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 537e26d840e948538b7ae7d47ad5e82d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681095359377 2023-04-10 02:56:02,098 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting efc43354fba74453babea5c9a887b483, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1681095360542 2023-04-10 02:56:02,106 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#C#compaction#451 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:02,160 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:02,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:02,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:02,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:02,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:02,163 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:02,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:02,163 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:02,164 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,164 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,170 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,170 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,170 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,170 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:02,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:02,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:02,172 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:02,172 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,172 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:02,172 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:02,172 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:02,172 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,172 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:02,172 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-10 02:56:02,172 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:02,172 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:02,173 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:02,173 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:02,173 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:02,173 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:02,181 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/75aa1dd627714233bd3b8f9e4d0462a6 2023-04-10 02:56:02,189 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/29b8287c9e464269bf4999655beff4d0 2023-04-10 02:56:02,198 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/42a9b0b992a34cb2a529024409b4bfb8 2023-04-10 02:56:02,200 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/75aa1dd627714233bd3b8f9e4d0462a6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/75aa1dd627714233bd3b8f9e4d0462a6 2023-04-10 02:56:02,203 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/75aa1dd627714233bd3b8f9e4d0462a6, entries=150, sequenceid=365, filesize=11.9 K 2023-04-10 02:56:02,204 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/29b8287c9e464269bf4999655beff4d0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/29b8287c9e464269bf4999655beff4d0 2023-04-10 02:56:02,207 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/29b8287c9e464269bf4999655beff4d0, entries=150, sequenceid=365, filesize=11.9 K 2023-04-10 02:56:02,207 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/42a9b0b992a34cb2a529024409b4bfb8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/42a9b0b992a34cb2a529024409b4bfb8 2023-04-10 02:56:02,210 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/42a9b0b992a34cb2a529024409b4bfb8, entries=150, sequenceid=365, filesize=11.9 K 2023-04-10 02:56:02,211 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=0 B/0 for 31f062e6d6eacf38c98ea51ae134e46f in 39ms, sequenceid=365, compaction requested=false 2023-04-10 02:56:02,211 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:02,211 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,211 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:02,211 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:02,211 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,211 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:02,211 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:02,213 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,213 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,213 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,213 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:02,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:02,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c045a70[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:02,215 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:02,216 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,217 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:02,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:02,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:02,217 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,217 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:02,218 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,218 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:02,218 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,219 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:02,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:02,221 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:02,221 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:02,221 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:02,221 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:02,221 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:02,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,224 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,224 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,225 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,227 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,227 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:02,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:02,227 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,227 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,228 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,228 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:02,228 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:02,227 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,228 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:02,228 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,228 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,228 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,328 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:02,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:02,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:02,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:02,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:02,331 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:02,331 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,331 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:02,331 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:02,332 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,332 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,333 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,333 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,333 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,333 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:02,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:02,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:02,335 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,335 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:02,335 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:02,335 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,335 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:02,335 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:02,335 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,335 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:02,335 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:02,336 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,336 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,337 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,337 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:02,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6210b926[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:02,339 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:02,339 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,345 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:02,345 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,345 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:02,345 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:02,346 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:02,347 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,347 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,347 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,347 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:02,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:02,349 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:02,349 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:02,349 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:02,349 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:02,349 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:02,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,351 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,351 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,352 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,356 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:02,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:02,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,356 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:02,356 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:02,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:02,446 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:56:02,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:02,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:02,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:02,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:02,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:02,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:02,456 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:02,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:02,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/ea1a82cbd9224c179a84fed8b1debe13 2023-04-10 02:56:02,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:02,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:02,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:02,460 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:02,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:02,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:02,461 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,461 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,463 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,463 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,464 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,464 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:02,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:02,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:02,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:02,465 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:02,465 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,465 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:02,476 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:02,476 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:02,476 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,476 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:02,476 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:02,476 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,479 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:02,479 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:02,481 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,481 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/7132a2807b83490fa8aaee39d23dd4cc 2023-04-10 02:56:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:02,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d0afbc4[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:02,483 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:02,483 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,484 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:02,485 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:02,485 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:02,485 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,485 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:02,486 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:02,486 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,486 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,486 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,490 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/058a3783cc774c4798f03ca74d2bf0fa as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/058a3783cc774c4798f03ca74d2bf0fa 2023-04-10 02:56:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:02,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:02,498 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:02,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:02,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:02,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:02,498 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:02,499 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,499 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,499 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,500 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,501 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,503 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,503 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,503 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:02,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:02,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,504 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:02,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,504 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:02,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,513 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/8a2254faeb60469585fdd4cc7d62cdc7 2023-04-10 02:56:02,517 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/A of 31f062e6d6eacf38c98ea51ae134e46f into 058a3783cc774c4798f03ca74d2bf0fa(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:02,517 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:02,517 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/A, priority=8, startTime=1681095362054; duration=0sec 2023-04-10 02:56:02,517 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:02,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/ea1a82cbd9224c179a84fed8b1debe13 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ea1a82cbd9224c179a84fed8b1debe13 2023-04-10 02:56:02,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1681095362479, max timestamp=1681095362503Num uniques 859; 2023-04-10 02:56:02,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=10484, suffix heap size=117064,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=229 2023-04-10 02:56:02,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095362479, max timestamp=1681095362503Num uniques 859; 2023-04-10 02:56:02,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:56:02,519 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/b2f671d8fc83485c8a67f00c7b9b4365 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/b2f671d8fc83485c8a67f00c7b9b4365 2023-04-10 02:56:02,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095362479, max timestamp=1681095362517Num uniques 859; 2023-04-10 02:56:02,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:56:02,523 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ea1a82cbd9224c179a84fed8b1debe13, entries=200, sequenceid=377, filesize=14.3 K 2023-04-10 02:56:02,523 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/7132a2807b83490fa8aaee39d23dd4cc as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7132a2807b83490fa8aaee39d23dd4cc 2023-04-10 02:56:02,525 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/C of 31f062e6d6eacf38c98ea51ae134e46f into b2f671d8fc83485c8a67f00c7b9b4365(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:02,525 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:02,525 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/C, priority=8, startTime=1681095362055; duration=0sec 2023-04-10 02:56:02,525 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:02,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7132a2807b83490fa8aaee39d23dd4cc, entries=150, sequenceid=377, filesize=11.9 K 2023-04-10 02:56:02,527 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/8a2254faeb60469585fdd4cc7d62cdc7 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/8a2254faeb60469585fdd4cc7d62cdc7 2023-04-10 02:56:02,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/8a2254faeb60469585fdd4cc7d62cdc7, entries=150, sequenceid=377, filesize=11.9 K 2023-04-10 02:56:02,530 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=47.86 KB/49010 for 31f062e6d6eacf38c98ea51ae134e46f in 84ms, sequenceid=377, compaction requested=true 2023-04-10 02:56:02,530 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:02,531 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:02,531 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:02,531 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:02,531 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:02,531 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:02,531 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:02,531 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:02,531 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/A is initiating minor compaction (all files) 2023-04-10 02:56:02,532 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/B is initiating minor compaction (all files) 2023-04-10 02:56:02,532 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/A in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,532 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/B in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,532 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/058a3783cc774c4798f03ca74d2bf0fa, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/75aa1dd627714233bd3b8f9e4d0462a6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ea1a82cbd9224c179a84fed8b1debe13] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=38.8 K 2023-04-10 02:56:02,532 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/5645a46c898947f9aca252deed0dbbfe, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/29b8287c9e464269bf4999655beff4d0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7132a2807b83490fa8aaee39d23dd4cc] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=36.4 K 2023-04-10 02:56:02,532 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 058a3783cc774c4798f03ca74d2bf0fa, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1681095359379 2023-04-10 02:56:02,532 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 5645a46c898947f9aca252deed0dbbfe, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1681095359379 2023-04-10 02:56:02,532 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 75aa1dd627714233bd3b8f9e4d0462a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681095361270 2023-04-10 02:56:02,532 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 29b8287c9e464269bf4999655beff4d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681095361270 2023-04-10 02:56:02,533 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ea1a82cbd9224c179a84fed8b1debe13, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681095362436 2023-04-10 02:56:02,533 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 7132a2807b83490fa8aaee39d23dd4cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681095362437 2023-04-10 02:56:02,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:02,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=54.57 KB heapSize=143.27 KB 2023-04-10 02:56:02,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:02,534 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:02,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:02,534 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:02,535 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:02,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:02,539 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#B#compaction#460 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:02,539 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#A#compaction#459 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:02,547 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.79 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/e7c34ec9e255401bb12c77efdafe9743 2023-04-10 02:56:02,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095422560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095422562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095362556, max timestamp=1681095362566Num uniques 859; 2023-04-10 02:56:02,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:02,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095362556, max timestamp=1681095362566Num uniques 859; 2023-04-10 02:56:02,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:02,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095362556, max timestamp=1681095362566Num uniques 859; 2023-04-10 02:56:02,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:02,604 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:02,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:02,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:02,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:02,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:02,607 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:02,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:02,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:02,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,609 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,609 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,609 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,609 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:02,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:02,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:02,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:02,611 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,611 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:02,611 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:02,611 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:02,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:02,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:02,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:02,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095422605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,613 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,613 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,613 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,613 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095422606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:02,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70173a73[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:02,615 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:02,616 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,617 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,617 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:02,617 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:02,617 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:02,617 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,617 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:02,618 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,618 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:02,618 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,618 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:02,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:02,620 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:02,621 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:02,621 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:02,621 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:02,621 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:02,622 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,622 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,622 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095422608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,626 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,629 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,629 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,629 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,629 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,629 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,629 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:02,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:02,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:02,629 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,630 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:02,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,630 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:02,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095422668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095422668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095422712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095422716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,730 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:02,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:02,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095422724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:02,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:02,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:02,733 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:02,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:02,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:02,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,733 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,735 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,735 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,735 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,735 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:02,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:02,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:02,736 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:02,736 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,736 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:02,736 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:02,736 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:02,736 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,736 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:02,736 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:02,737 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,737 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:02,737 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:02,737 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,737 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:02,737 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:02,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,739 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,739 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,739 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:02,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e1e29db[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:02,741 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:02,741 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,742 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,742 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:02,742 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:02,742 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:02,742 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,742 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:02,743 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:02,743 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,743 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,743 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:02,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:02,745 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:02,745 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:02,745 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:02,745 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:02,745 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:02,750 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,750 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,750 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,751 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,752 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,754 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,754 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,754 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:02,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:02,754 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,754 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:02,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,755 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:02,755 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:02,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,855 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:02,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:02,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:02,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:02,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:02,858 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:02,858 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,858 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:02,858 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:02,859 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,859 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,860 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:02,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:02,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:02,861 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:02,861 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,861 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:02,861 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:02,862 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,862 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:02,862 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:02,862 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,863 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,864 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,864 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:02,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49207cb7[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:02,865 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:02,865 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,866 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,866 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:02,866 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:02,866 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,867 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:02,866 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:02,869 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:02,869 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,869 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,869 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:02,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:02,871 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:02,871 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:02,871 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:02,871 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:02,871 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:02,873 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,873 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,873 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,874 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,875 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,878 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,878 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,878 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,878 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,878 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,878 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,878 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,878 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,878 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:02,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:02,878 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:02,878 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:02,878 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:02,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095422875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095422878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095422923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095422923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:02,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095422933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,949 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/c7272ae278c24ceab0630cdc3de2710b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/c7272ae278c24ceab0630cdc3de2710b 2023-04-10 02:56:02,950 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/36c9004b217b497782dcae22f9f83620 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/36c9004b217b497782dcae22f9f83620 2023-04-10 02:56:02,953 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/A of 31f062e6d6eacf38c98ea51ae134e46f into 36c9004b217b497782dcae22f9f83620(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:02,954 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:02,954 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/A, priority=13, startTime=1681095362531; duration=0sec 2023-04-10 02:56:02,954 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:02,954 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:02,954 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/B of 31f062e6d6eacf38c98ea51ae134e46f into c7272ae278c24ceab0630cdc3de2710b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:02,954 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:02,954 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/B, priority=13, startTime=1681095362531; duration=0sec 2023-04-10 02:56:02,954 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:02,955 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:02,955 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/C is initiating minor compaction (all files) 2023-04-10 02:56:02,955 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/C in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,955 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/b2f671d8fc83485c8a67f00c7b9b4365, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/42a9b0b992a34cb2a529024409b4bfb8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/8a2254faeb60469585fdd4cc7d62cdc7] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=36.4 K 2023-04-10 02:56:02,955 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting b2f671d8fc83485c8a67f00c7b9b4365, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1681095359379 2023-04-10 02:56:02,956 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 42a9b0b992a34cb2a529024409b4bfb8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681095361270 2023-04-10 02:56:02,956 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 8a2254faeb60469585fdd4cc7d62cdc7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681095362437 2023-04-10 02:56:02,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/2d63a2c2a8ef446890cd232c2ad9c951 2023-04-10 02:56:02,961 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#C#compaction#462 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:02,979 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:02,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:02,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:02,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:02,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:02,982 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:02,982 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,982 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:02,982 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:02,982 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,982 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,984 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:02,984 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:02,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:02,984 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,984 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:02,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:02,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:02,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:02,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:02,986 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:02,986 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:02,986 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,987 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:02,987 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:02,987 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:02,987 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:02,987 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:02,987 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,987 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:02,987 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:02,989 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:02,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@231e6a8c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:02,991 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:02,991 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:02,992 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:02,992 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:02,992 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:02,992 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,992 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:02,993 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,993 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,993 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:02,994 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:02,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:02,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:02,996 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:02,996 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:02,996 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:02,996 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:02,996 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:02,997 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,997 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:02,997 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:02,998 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:02,999 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,002 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,002 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:03,002 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,002 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:03,002 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:03,002 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,002 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,002 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,002 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:03,002 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:03,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:03,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:03,002 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:03,002 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:03,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:03,103 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:03,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:03,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:03,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:03,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:03,106 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:03,106 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:03,106 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:03,106 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:03,109 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,109 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,110 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,110 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:03,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:03,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,111 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:03,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:03,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:03,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:03,112 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:03,112 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:03,112 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:03,112 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:03,112 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:03,112 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:03,112 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:03,112 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:03,112 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:03,112 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:03,112 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:03,112 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,113 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:03,113 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:03,114 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,114 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:03,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:03,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:03,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a93083e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:03,116 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:03,116 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,117 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,118 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:03,118 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:03,118 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:03,119 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,119 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:03,120 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:03,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:03,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:03,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:03,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:03,123 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:03,123 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:03,123 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:03,123 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:03,123 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:03,125 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,125 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:03,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,125 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:03,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:03,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:03,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:03,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:03,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,127 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:03,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:03,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:03,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,138 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:03,138 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:03,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:03,138 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,138 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,138 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,138 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:03,138 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:03,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:03,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:03,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095423185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:03,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095423185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,238 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:03,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:03,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:03,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095423233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:03,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:03,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095423233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:03,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095423239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:03,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:03,242 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:03,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:03,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:03,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:03,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,245 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:03,245 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:03,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:03,245 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,245 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:03,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:03,246 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:03,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:03,246 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:03,246 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:03,246 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:03,246 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:03,246 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:03,247 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:03,247 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:03,247 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:03,247 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:03,248 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:03,249 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,249 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@351ca446[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:03,250 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:03,250 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,251 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,251 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:03,251 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:03,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:03,251 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:03,252 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:03,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:03,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:03,253 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:03,257 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:03,257 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:03,257 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:03,257 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:03,257 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:03,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:03,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,259 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:03,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,260 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,261 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,263 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,263 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,263 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:03,263 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:03,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:03,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:03,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:03,263 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:03,263 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,264 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,264 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,264 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,264 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:03,264 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:03,264 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:03,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:03,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:03,364 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:03,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:03,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:03,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:03,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:03,368 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:03,368 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:03,368 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:03,368 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:03,370 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,370 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,371 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:03,371 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,372 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,372 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:03,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:03,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:03,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:03,385 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/ea7ddf745230464e8c5932c998879ddd 2023-04-10 02:56:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:03,385 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:03,385 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:03,386 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:03,386 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:03,387 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,387 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:03,388 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,388 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:03,390 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/e7c34ec9e255401bb12c77efdafe9743 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/e7c34ec9e255401bb12c77efdafe9743 2023-04-10 02:56:03,392 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/ec421356b2b64971a9d0fd558a3ccc21 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/ec421356b2b64971a9d0fd558a3ccc21 2023-04-10 02:56:03,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:03,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f13b396[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:03,394 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:03,394 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,396 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,396 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/e7c34ec9e255401bb12c77efdafe9743, entries=200, sequenceid=403, filesize=14.3 K 2023-04-10 02:56:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:03,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:03,397 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/C of 31f062e6d6eacf38c98ea51ae134e46f into ec421356b2b64971a9d0fd558a3ccc21(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:03,397 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:03,397 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/C, priority=13, startTime=1681095362531; duration=0sec 2023-04-10 02:56:03,397 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/2d63a2c2a8ef446890cd232c2ad9c951 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2d63a2c2a8ef446890cd232c2ad9c951 2023-04-10 02:56:03,397 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:03,398 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:03,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:03,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:03,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:03,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:03,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:03,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:03,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:03,402 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:03,402 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:03,402 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:03,402 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:03,402 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:03,404 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,404 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:03,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,404 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:03,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:03,404 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,406 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:03,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,409 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:03,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:03,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:03,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:03,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:03,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,410 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,410 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,410 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:03,410 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,410 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:03,410 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:03,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:03,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2d63a2c2a8ef446890cd232c2ad9c951, entries=150, sequenceid=403, filesize=11.9 K 2023-04-10 02:56:03,412 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/ea7ddf745230464e8c5932c998879ddd as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/ea7ddf745230464e8c5932c998879ddd 2023-04-10 02:56:03,417 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/ea7ddf745230464e8c5932c998879ddd, entries=150, sequenceid=403, filesize=11.9 K 2023-04-10 02:56:03,418 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~54.57 KB/55880, heapSize ~143.22 KB/146656, currentSize=134.18 KB/137400 for 31f062e6d6eacf38c98ea51ae134e46f in 884ms, sequenceid=403, compaction requested=false 2023-04-10 02:56:03,418 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:03,510 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:03,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:03,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:03,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:03,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:03,513 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:03,513 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:03,514 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:03,514 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:03,514 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,514 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,515 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,515 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:03,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:03,515 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,515 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:03,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:03,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:03,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:03,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:03,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:03,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:03,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:03,517 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:03,517 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:03,517 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-10 02:56:03,518 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:03,518 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:03,518 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:03,518 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:03,518 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:03,518 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:03,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:03,695 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:03,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:03,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095423712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:03,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095423714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:03,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095423742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:03,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095423742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:03,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095423744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:03,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095423819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:03,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095423825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:03,925 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/0052d694695f4133b4de53e45dde6d5a 2023-04-10 02:56:03,941 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/7db9897eef74447d921bd4dc5dec3147 2023-04-10 02:56:04,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:04,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095424024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:04,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095424031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:04,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095424335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:04,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095424343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,378 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/e364c419efaf48f0823479a2ad8c0e67 2023-04-10 02:56:04,382 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/0052d694695f4133b4de53e45dde6d5a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/0052d694695f4133b4de53e45dde6d5a 2023-04-10 02:56:04,385 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/0052d694695f4133b4de53e45dde6d5a, entries=150, sequenceid=443, filesize=11.9 K 2023-04-10 02:56:04,385 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/7db9897eef74447d921bd4dc5dec3147 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7db9897eef74447d921bd4dc5dec3147 2023-04-10 02:56:04,388 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7db9897eef74447d921bd4dc5dec3147, entries=150, sequenceid=443, filesize=11.9 K 2023-04-10 02:56:04,389 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/e364c419efaf48f0823479a2ad8c0e67 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/e364c419efaf48f0823479a2ad8c0e67 2023-04-10 02:56:04,392 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/e364c419efaf48f0823479a2ad8c0e67, entries=150, sequenceid=443, filesize=11.9 K 2023-04-10 02:56:04,392 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 31f062e6d6eacf38c98ea51ae134e46f in 875ms, sequenceid=443, compaction requested=true 2023-04-10 02:56:04,392 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:04,392 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:04,392 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:04,392 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:04,393 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,393 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:04,393 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:04,394 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:04,394 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:04,394 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:04,394 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:04,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:04,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f89fcf2[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:04,396 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:04,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:04,398 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:04,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:04,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:04,398 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,398 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:04,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:04,399 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:04,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:04,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:04,401 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:04,401 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:04,401 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:04,401 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:04,402 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:04,403 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:04,403 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:04,404 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:04,404 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,409 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:04,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:04,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:04,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:04,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:04,413 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:04,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:04,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:04,414 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:04,414 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:04,416 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:04,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:04,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:04,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:04,514 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:04,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:04,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:04,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:04,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:04,520 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:04,520 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:04,520 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:04,520 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:04,521 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:04,521 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:04,522 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:04,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:04,522 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:04,524 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,524 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:04,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:04,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:04,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:04,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:04,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:04,536 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:04,536 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:04,536 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:04,536 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:04,536 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:56:04,536 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:04,536 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:04,537 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:04,537 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:04,537 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:04,537 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:04,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:04,749 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:04,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095364775, max timestamp=1681095364787Num uniques 859; 2023-04-10 02:56:04,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095364775, max timestamp=1681095364787Num uniques 859; 2023-04-10 02:56:04,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095364775, max timestamp=1681095364787Num uniques 859; 2023-04-10 02:56:04,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:04,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:04,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:04,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:04,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095424819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:04,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095424820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:04,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095424821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:04,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095424845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:04,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095424848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:04,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095424922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:04,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095424934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:04,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095424935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:04,962 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/e8360a4beebd433f81eeb97b21d4db65 2023-04-10 02:56:05,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095425125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095425139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095425140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,372 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/45ef2d90e6c3456cbb151cad2a852d74 2023-04-10 02:56:05,381 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/6c8b8518f1c5427a9f1e16e92c55758d 2023-04-10 02:56:05,384 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/e8360a4beebd433f81eeb97b21d4db65 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/e8360a4beebd433f81eeb97b21d4db65 2023-04-10 02:56:05,387 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/e8360a4beebd433f81eeb97b21d4db65, entries=150, sequenceid=453, filesize=11.9 K 2023-04-10 02:56:05,388 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/45ef2d90e6c3456cbb151cad2a852d74 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/45ef2d90e6c3456cbb151cad2a852d74 2023-04-10 02:56:05,390 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/45ef2d90e6c3456cbb151cad2a852d74, entries=150, sequenceid=453, filesize=11.9 K 2023-04-10 02:56:05,391 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/6c8b8518f1c5427a9f1e16e92c55758d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/6c8b8518f1c5427a9f1e16e92c55758d 2023-04-10 02:56:05,394 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/6c8b8518f1c5427a9f1e16e92c55758d, entries=150, sequenceid=453, filesize=11.9 K 2023-04-10 02:56:05,395 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 31f062e6d6eacf38c98ea51ae134e46f in 858ms, sequenceid=453, compaction requested=true 2023-04-10 02:56:05,395 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:05,395 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:05,395 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:05,395 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:05,395 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,395 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:05,395 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:05,396 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,396 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:05,397 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,397 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:05,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:05,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5da9c3c5[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:05,398 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:05,398 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,399 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:05,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:05,399 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:05,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:05,401 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:05,401 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:05,401 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,401 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:05,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:05,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:05,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:05,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:05,403 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:05,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:05,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:05,403 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:05,403 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:05,405 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,405 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:05,405 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:05,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:05,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:05,406 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,407 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:05,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:05,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:05,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:05,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:05,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,409 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,409 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:05,409 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:05,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:05,410 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:05,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:05,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:05,438 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-10 02:56:05,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:05,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:56:05,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:05,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:56:05,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:05,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:56:05,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095425455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095425455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095425458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,510 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:05,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:05,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:05,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:05,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:05,513 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:05,513 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:05,513 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:05,513 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:05,513 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,513 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,515 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:05,515 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:05,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:05,515 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,515 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:05,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:05,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:05,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:05,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:05,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:05,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:05,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:05,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:05,516 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:05,516 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:05,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:05,516 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:05,517 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:05,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:05,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:05,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:05,517 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:05,518 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,518 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:05,518 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:05,518 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:05,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:05,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:05,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:05,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dc762a2[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:05,520 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:05,520 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,522 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,522 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:05,522 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:05,522 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:05,522 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,523 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:05,524 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,524 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:05,524 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:05,524 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:05,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:05,526 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:05,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:05,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:05,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:05,526 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:05,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,528 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:05,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:05,528 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,529 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,531 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,531 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,532 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,532 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:05,532 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:05,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:05,532 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:05,532 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:05,532 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,532 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:05,532 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,532 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:05,532 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:05,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:05,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095425564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095425564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095425564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,632 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:05,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:05,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:05,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:05,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:05,635 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:05,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:05,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:05,636 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:05,636 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,636 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,638 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,638 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:05,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:05,638 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,638 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:05,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:05,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:05,639 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:05,639 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:05,639 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:05,640 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:05,640 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:05,640 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:05,640 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:05,640 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:05,640 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:05,640 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:05,640 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:05,640 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,640 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:05,640 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:05,641 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:05,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:05,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:05,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:05,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6faaef8f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:05,643 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:05,643 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,644 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,645 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:05,645 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:05,645 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:05,645 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,645 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:05,646 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:05,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:05,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:05,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:05,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:05,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:05,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:05,650 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:05,651 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:05,651 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:05,651 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:05,651 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:05,653 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,653 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:05,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,653 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:05,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:05,654 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,657 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:05,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,660 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:05,660 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:05,660 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,660 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:05,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:05,660 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:05,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:05,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:05,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:05,661 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:05,661 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:05,761 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:05,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:05,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:05,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:05,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:05,768 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:05,768 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:05,768 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:05,768 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:05,773 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,774 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,775 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:05,775 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:05,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:05,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:05,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:05,777 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:05,777 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:05,777 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:05,777 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:05,777 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:05,777 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:05,777 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:05,777 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:05,777 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:05,777 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:05,777 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:05,777 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,778 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:05,778 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:05,778 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,778 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:05,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095425771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095425772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095425773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,787 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,787 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:05,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,788 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:05,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:05,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f24f0e4[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:05,789 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:05,789 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,791 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:05,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:05,791 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:05,791 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,791 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:05,792 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:05,792 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,792 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:05,793 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:05,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:05,795 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:05,795 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:05,795 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:05,795 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:05,795 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:05,798 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,798 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:05,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:05,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,798 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:05,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:05,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:05,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:05,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:05,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:05,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,801 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:05,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:05,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:05,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:05,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:05,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:05,804 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:05,805 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:05,804 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,805 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,805 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,805 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,805 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:05,805 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,805 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:05,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:05,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/f0667e9d92984672bddb84c8d3373ec2 2023-04-10 02:56:05,866 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/d8631e4bdfd54e73bc10e8cc437b1342 2023-04-10 02:56:05,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095425858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:05,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095425864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,886 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/7513ae5f48f641baa6eac93cf52a142d 2023-04-10 02:56:05,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/f0667e9d92984672bddb84c8d3373ec2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/f0667e9d92984672bddb84c8d3373ec2 2023-04-10 02:56:05,892 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/f0667e9d92984672bddb84c8d3373ec2, entries=200, sequenceid=491, filesize=14.3 K 2023-04-10 02:56:05,893 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/d8631e4bdfd54e73bc10e8cc437b1342 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d8631e4bdfd54e73bc10e8cc437b1342 2023-04-10 02:56:05,896 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d8631e4bdfd54e73bc10e8cc437b1342, entries=150, sequenceid=491, filesize=11.9 K 2023-04-10 02:56:05,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/7513ae5f48f641baa6eac93cf52a142d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/7513ae5f48f641baa6eac93cf52a142d 2023-04-10 02:56:05,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/7513ae5f48f641baa6eac93cf52a142d, entries=150, sequenceid=491, filesize=11.9 K 2023-04-10 02:56:05,901 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 31f062e6d6eacf38c98ea51ae134e46f in 463ms, sequenceid=491, compaction requested=true 2023-04-10 02:56:05,901 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:05,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:05,901 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:56:05,901 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:56:05,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:05,902 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:05,902 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66662 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:56:05,902 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:56:05,903 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/A is initiating minor compaction (all files) 2023-04-10 02:56:05,903 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/B is initiating minor compaction (all files) 2023-04-10 02:56:05,903 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/A in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:05,903 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/B in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:05,903 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/36c9004b217b497782dcae22f9f83620, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/e7c34ec9e255401bb12c77efdafe9743, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/0052d694695f4133b4de53e45dde6d5a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/e8360a4beebd433f81eeb97b21d4db65, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/f0667e9d92984672bddb84c8d3373ec2] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=65.1 K 2023-04-10 02:56:05,903 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/c7272ae278c24ceab0630cdc3de2710b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2d63a2c2a8ef446890cd232c2ad9c951, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7db9897eef74447d921bd4dc5dec3147, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/45ef2d90e6c3456cbb151cad2a852d74, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d8631e4bdfd54e73bc10e8cc437b1342] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=60.3 K 2023-04-10 02:56:05,903 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 36c9004b217b497782dcae22f9f83620, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681095362437 2023-04-10 02:56:05,903 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting c7272ae278c24ceab0630cdc3de2710b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681095362437 2023-04-10 02:56:05,903 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting e7c34ec9e255401bb12c77efdafe9743, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681095362479 2023-04-10 02:56:05,903 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 2d63a2c2a8ef446890cd232c2ad9c951, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681095362479 2023-04-10 02:56:05,904 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 0052d694695f4133b4de53e45dde6d5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681095362604 2023-04-10 02:56:05,904 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 7db9897eef74447d921bd4dc5dec3147, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681095362604 2023-04-10 02:56:05,904 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting e8360a4beebd433f81eeb97b21d4db65, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681095363700 2023-04-10 02:56:05,904 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 45ef2d90e6c3456cbb151cad2a852d74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681095363700 2023-04-10 02:56:05,904 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting f0667e9d92984672bddb84c8d3373ec2, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1681095364817 2023-04-10 02:56:05,905 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d8631e4bdfd54e73bc10e8cc437b1342, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1681095364817 2023-04-10 02:56:05,905 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:05,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:05,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:05,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:05,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:05,909 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:05,909 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:05,909 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:05,909 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:05,910 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,910 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,911 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,911 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:05,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:05,911 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:05,911 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:05,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:05,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:05,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:05,912 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:05,912 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:05,913 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:05,913 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:05,913 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:05,913 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:05,913 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:05,913 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-10 02:56:05,913 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:05,913 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:05,913 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:05,913 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:05,913 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:05,914 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:05,914 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#A#compaction#473 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:05,917 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#B#compaction#474 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:05,923 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/f93da79f667a4e109baebd7c4b7ec6cc 2023-04-10 02:56:06,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:06,084 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:06,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095366107, max timestamp=1681095366119Num uniques 859; 2023-04-10 02:56:06,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095366107, max timestamp=1681095366119Num uniques 859; 2023-04-10 02:56:06,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681095366107, max timestamp=1681095366119Num uniques 859; 2023-04-10 02:56:06,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:56:06,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:56:06,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-10 02:56:06,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095426150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095426154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095426155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095426260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095426262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095426264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,328 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/a7ccd70c93a745588296c6efd8f9e6a3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/a7ccd70c93a745588296c6efd8f9e6a3 2023-04-10 02:56:06,330 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/efeded2691b94991969c9105b06a6b35 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/efeded2691b94991969c9105b06a6b35 2023-04-10 02:56:06,331 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/de47b335bdcc4526a89f772c5933adcc 2023-04-10 02:56:06,332 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/A of 31f062e6d6eacf38c98ea51ae134e46f into a7ccd70c93a745588296c6efd8f9e6a3(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:06,332 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:06,332 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/A, priority=11, startTime=1681095365901; duration=0sec 2023-04-10 02:56:06,332 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:06,332 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:56:06,335 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:56:06,335 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/C is initiating minor compaction (all files) 2023-04-10 02:56:06,335 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/C in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:06,335 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/ec421356b2b64971a9d0fd558a3ccc21, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/ea7ddf745230464e8c5932c998879ddd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/e364c419efaf48f0823479a2ad8c0e67, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/6c8b8518f1c5427a9f1e16e92c55758d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/7513ae5f48f641baa6eac93cf52a142d] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=60.3 K 2023-04-10 02:56:06,336 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ec421356b2b64971a9d0fd558a3ccc21, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681095362437 2023-04-10 02:56:06,336 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/B of 31f062e6d6eacf38c98ea51ae134e46f into efeded2691b94991969c9105b06a6b35(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:06,336 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:06,336 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/B, priority=11, startTime=1681095365901; duration=0sec 2023-04-10 02:56:06,336 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:06,336 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ea7ddf745230464e8c5932c998879ddd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681095362479 2023-04-10 02:56:06,338 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting e364c419efaf48f0823479a2ad8c0e67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681095362604 2023-04-10 02:56:06,338 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 6c8b8518f1c5427a9f1e16e92c55758d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681095363700 2023-04-10 02:56:06,338 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 7513ae5f48f641baa6eac93cf52a142d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1681095364817 2023-04-10 02:56:06,345 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#C#compaction#478 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:06,349 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/faa0d787125642b7921c8d764b8d9a4f 2023-04-10 02:56:06,353 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/f93da79f667a4e109baebd7c4b7ec6cc as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/f93da79f667a4e109baebd7c4b7ec6cc 2023-04-10 02:56:06,357 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/f93da79f667a4e109baebd7c4b7ec6cc, entries=150, sequenceid=500, filesize=11.9 K 2023-04-10 02:56:06,357 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/de47b335bdcc4526a89f772c5933adcc as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/de47b335bdcc4526a89f772c5933adcc 2023-04-10 02:56:06,361 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/de47b335bdcc4526a89f772c5933adcc, entries=150, sequenceid=500, filesize=11.9 K 2023-04-10 02:56:06,362 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/faa0d787125642b7921c8d764b8d9a4f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/faa0d787125642b7921c8d764b8d9a4f 2023-04-10 02:56:06,365 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/faa0d787125642b7921c8d764b8d9a4f, entries=150, sequenceid=500, filesize=11.9 K 2023-04-10 02:56:06,365 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 31f062e6d6eacf38c98ea51ae134e46f in 452ms, sequenceid=500, compaction requested=false 2023-04-10 02:56:06,365 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:06,365 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:06,366 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:06,366 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:06,366 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,366 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:06,366 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:06,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,368 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:06,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:06,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c80ad2e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:06,369 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:06,369 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,370 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:06,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:06,370 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,370 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:06,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:06,372 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:06,372 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:06,372 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,373 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:06,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:06,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:06,375 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:06,375 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:06,376 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:06,376 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:06,376 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:06,377 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,377 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,377 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:06,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:06,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:06,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,378 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,379 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:06,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:06,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:06,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:06,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:06,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:06,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,381 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:06,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:06,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,381 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,381 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:06,382 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:06,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:06,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:06,469 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-10 02:56:06,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:06,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:06,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:06,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:06,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:06,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:06,482 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:06,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:06,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:06,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:06,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:06,484 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:06,485 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:06,485 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:06,485 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:06,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,485 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,485 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095426480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,486 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,486 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:06,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:06,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:06,487 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,487 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:06,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095426485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095426485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:06,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:06,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:06,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:06,488 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:06,488 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:06,488 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:06,488 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:06,488 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:06,488 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:06,488 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:06,488 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:06,488 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:06,489 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:06,489 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:06,489 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,489 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:06,489 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:06,490 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,491 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,491 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,491 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:06,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:06,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a640e6e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:06,492 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:06,492 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,493 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:06,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:06,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:06,494 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,494 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:06,495 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:06,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:06,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,496 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:06,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:06,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:06,498 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:06,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:06,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:06,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:06,498 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:06,499 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,499 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,499 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:06,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,500 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,501 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:06,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:06,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:06,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:06,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:06,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:06,504 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:06,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:06,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:06,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:06,504 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:06,504 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:06,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095426586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095426588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095426588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,605 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:06,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:06,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:06,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:06,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:06,607 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:06,607 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:06,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:06,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:06,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,609 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:06,609 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:06,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:06,610 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,610 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:06,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:06,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:06,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:06,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:06,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:06,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:06,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:06,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:06,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:06,611 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:06,611 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:06,611 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:06,611 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:06,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:06,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:06,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:06,611 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:06,613 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,613 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,613 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,613 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:06,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@120a4a3a[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:06,614 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:06,614 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,619 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:06,619 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,619 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:06,620 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,620 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:06,620 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:06,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,620 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:06,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:06,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:06,622 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:06,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:06,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:06,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:06,622 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:06,624 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,624 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,624 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:06,624 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:06,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:06,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,625 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,629 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:06,629 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,629 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:06,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:06,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:06,629 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,629 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:06,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:06,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:06,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:06,630 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:06,630 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:06,629 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:06,630 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:06,730 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:06,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:06,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:06,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:06,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:06,736 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:06,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:06,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:06,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:06,736 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,737 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,738 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,738 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,738 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:06,739 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:06,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:06,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:06,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:06,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:06,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:06,741 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:06,741 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:06,741 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:06,741 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:06,741 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:06,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:06,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:06,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:06,741 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:06,742 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:06,742 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:06,742 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,742 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:06,742 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:06,743 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,743 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,743 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:06,744 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:06,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a812b4f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:06,750 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:06,751 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,752 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,752 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,752 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:06,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,752 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:06,752 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:06,752 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:06,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:06,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:06,754 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:06,754 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:06,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:06,757 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:06,757 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:06,757 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:06,757 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:06,757 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:06,759 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,759 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:06,759 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,760 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:06,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:06,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,761 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,761 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/c9dc8a08503f4e5ebb0b627d501e074e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/c9dc8a08503f4e5ebb0b627d501e074e 2023-04-10 02:56:06,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,763 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,763 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,763 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,763 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,763 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,763 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:06,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:06,763 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:06,764 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:06,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:06,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:06,763 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,764 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:06,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:06,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:06,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:06,764 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:06,764 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:06,766 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/C of 31f062e6d6eacf38c98ea51ae134e46f into c9dc8a08503f4e5ebb0b627d501e074e(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:06,766 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:06,766 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/C, priority=11, startTime=1681095365902; duration=0sec 2023-04-10 02:56:06,766 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:06,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095426791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095426791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:06,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095426792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,865 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:06,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:06,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:06,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:06,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:06,867 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:06,868 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:06,868 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:06,868 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:06,868 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,868 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,869 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,869 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:06,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:06,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:06,869 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,869 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:06,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:06,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:06,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:06,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:06,870 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:06,870 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:06,871 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:06,871 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:06,871 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:06,871 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:06,871 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:06,871 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:06,871 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:06,871 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:06,871 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:06,871 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,871 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:06,871 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:06,874 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,874 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,874 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,874 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:06,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:06,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@237108ec[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:06,875 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:06,875 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,876 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,877 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,877 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:06,877 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:06,877 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:06,877 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:06,877 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/a7eb6b4c3e5b4f0baa89fd348a351dc5 2023-04-10 02:56:06,878 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:06,879 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:06,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:06,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:06,882 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:06,882 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:06,882 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:06,882 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:06,882 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:06,885 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,885 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,885 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:06,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:06,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:06,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:06,886 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,887 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,890 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,891 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,891 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:06,891 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,891 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,891 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,891 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:06,891 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,891 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:06,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:06,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:06,891 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:06,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:06,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:06,892 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:06,892 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:06,992 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:06,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:06,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:06,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:06,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:06,995 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:06,995 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:06,995 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:06,995 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:06,996 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,996 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,997 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:06,997 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:06,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:06,997 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,997 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:06,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:06,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:06,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:06,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:06,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:06,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:06,999 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:06,999 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:06,999 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:06,999 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:07,000 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,001 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,001 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,001 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:07,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:07,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@296c7592[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:07,002 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:07,002 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,003 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,003 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:07,003 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:07,003 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,004 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:07,003 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:07,008 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:07,008 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,008 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,009 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:07,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:07,011 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:07,011 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:07,011 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:07,011 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:07,011 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:07,012 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,012 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,012 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,013 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,014 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,017 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:07,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:07,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:07,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,017 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,017 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:07,017 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:07,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:07,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095427106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:07,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095427106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:07,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095427106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,118 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:07,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:07,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:07,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:07,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:07,121 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:07,121 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,121 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:07,121 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:07,121 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,121 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,124 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,124 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,124 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:07,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:07,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:07,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:07,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:07,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:07,125 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,125 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:07,126 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:07,126 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,126 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:07,126 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:07,126 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,126 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:07,126 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:07,127 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,127 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:07,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:07,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f7a11f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:07,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:07,129 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,130 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,130 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:07,130 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:07,130 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:07,130 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,130 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:07,132 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,132 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:07,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:07,134 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:07,134 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:07,134 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:07,134 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:07,134 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:07,136 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,136 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,136 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,137 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,138 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:07,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:07,145 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,145 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,145 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:07,145 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:07,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,245 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:07,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:07,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:07,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:07,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:07,250 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:07,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:07,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:07,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,251 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,252 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,253 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,253 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,253 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:07,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:07,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:07,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:07,254 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,254 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:07,255 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:07,255 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:07,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:07,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:07,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:07,258 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,258 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,258 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:07,258 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:07,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a9f243d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:07,260 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:07,260 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,262 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,263 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:07,263 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:07,263 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:07,263 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,263 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:07,264 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,264 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:07,264 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,264 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:07,267 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:07,267 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:07,267 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:07,267 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:07,267 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:07,268 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,268 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,268 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,269 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,270 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:07,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:07,272 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,272 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:07,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,272 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:07,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,272 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:07,289 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/cec03427b33a48028cd983eb1e281fb0 2023-04-10 02:56:07,298 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/56ed5d05baea41e3b534afd1d7455cbb 2023-04-10 02:56:07,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/a7eb6b4c3e5b4f0baa89fd348a351dc5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/a7eb6b4c3e5b4f0baa89fd348a351dc5 2023-04-10 02:56:07,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/a7eb6b4c3e5b4f0baa89fd348a351dc5, entries=200, sequenceid=539, filesize=14.3 K 2023-04-10 02:56:07,305 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/cec03427b33a48028cd983eb1e281fb0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/cec03427b33a48028cd983eb1e281fb0 2023-04-10 02:56:07,308 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/cec03427b33a48028cd983eb1e281fb0, entries=150, sequenceid=539, filesize=11.9 K 2023-04-10 02:56:07,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/56ed5d05baea41e3b534afd1d7455cbb as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/56ed5d05baea41e3b534afd1d7455cbb 2023-04-10 02:56:07,311 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/56ed5d05baea41e3b534afd1d7455cbb, entries=150, sequenceid=539, filesize=11.9 K 2023-04-10 02:56:07,312 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=40.25 KB/41220 for 31f062e6d6eacf38c98ea51ae134e46f in 843ms, sequenceid=539, compaction requested=true 2023-04-10 02:56:07,312 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:07,312 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:07,312 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:07,312 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:07,312 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:07,312 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:07,313 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:07,313 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:07,313 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/A is initiating minor compaction (all files) 2023-04-10 02:56:07,313 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/B is initiating minor compaction (all files) 2023-04-10 02:56:07,313 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/A in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,313 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/B in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,313 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/a7ccd70c93a745588296c6efd8f9e6a3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/f93da79f667a4e109baebd7c4b7ec6cc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/a7eb6b4c3e5b4f0baa89fd348a351dc5] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=39.1 K 2023-04-10 02:56:07,313 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/efeded2691b94991969c9105b06a6b35, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/de47b335bdcc4526a89f772c5933adcc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/cec03427b33a48028cd983eb1e281fb0] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=36.7 K 2023-04-10 02:56:07,314 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting a7ccd70c93a745588296c6efd8f9e6a3, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1681095364817 2023-04-10 02:56:07,314 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting efeded2691b94991969c9105b06a6b35, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1681095364817 2023-04-10 02:56:07,314 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting f93da79f667a4e109baebd7c4b7ec6cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1681095365454 2023-04-10 02:56:07,314 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting de47b335bdcc4526a89f772c5933adcc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1681095365454 2023-04-10 02:56:07,314 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting a7eb6b4c3e5b4f0baa89fd348a351dc5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681095366143 2023-04-10 02:56:07,314 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting cec03427b33a48028cd983eb1e281fb0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681095366143 2023-04-10 02:56:07,320 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#A#compaction#482 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:07,320 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#B#compaction#483 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:07,373 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:07,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:07,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:07,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:07,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:07,376 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:07,376 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,376 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:07,376 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:07,376 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,377 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,379 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,379 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,379 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,379 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:07,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:07,380 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:07,380 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,380 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:07,380 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:07,380 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:07,380 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,380 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:07,380 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-10 02:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:07,387 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/022c174ef35a413d98727356a599fc35 2023-04-10 02:56:07,398 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/9a43baae2fd4473f9ee4a1f1d49e506c 2023-04-10 02:56:07,408 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/dc66a732b33242f58e067e5f33c2bc3a 2023-04-10 02:56:07,411 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/022c174ef35a413d98727356a599fc35 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/022c174ef35a413d98727356a599fc35 2023-04-10 02:56:07,414 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/022c174ef35a413d98727356a599fc35, entries=150, sequenceid=549, filesize=11.9 K 2023-04-10 02:56:07,415 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/9a43baae2fd4473f9ee4a1f1d49e506c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/9a43baae2fd4473f9ee4a1f1d49e506c 2023-04-10 02:56:07,418 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/9a43baae2fd4473f9ee4a1f1d49e506c, entries=150, sequenceid=549, filesize=11.9 K 2023-04-10 02:56:07,419 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/dc66a732b33242f58e067e5f33c2bc3a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/dc66a732b33242f58e067e5f33c2bc3a 2023-04-10 02:56:07,422 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/dc66a732b33242f58e067e5f33c2bc3a, entries=150, sequenceid=549, filesize=11.9 K 2023-04-10 02:56:07,423 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 31f062e6d6eacf38c98ea51ae134e46f in 43ms, sequenceid=549, compaction requested=true 2023-04-10 02:56:07,423 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:07,423 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,423 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:07,423 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:07,423 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,423 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:07,423 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:07,425 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:07,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@718261ea[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:07,427 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:07,427 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,428 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:07,428 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:07,428 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:07,428 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,428 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:07,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:07,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,429 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:07,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:07,432 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:07,432 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:07,432 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:07,432 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:07,432 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:07,434 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,435 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,435 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,438 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,439 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,444 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,444 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,444 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,444 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:07,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:07,444 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,444 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,444 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,444 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,444 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,445 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,445 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:07,445 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:07,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:07,545 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:07,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:07,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:07,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:07,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:07,548 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:07,548 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,548 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:07,548 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:07,548 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,548 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,549 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,549 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,550 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,550 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:07,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:07,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:07,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:07,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:07,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:07,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:07,551 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,551 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:07,551 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:07,551 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:07,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:07,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:07,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:07,553 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,553 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,554 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,554 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:07,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b885aad[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:07,555 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:07,555 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,556 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,557 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:07,557 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:07,557 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:07,557 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,557 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:07,558 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,558 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,558 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,558 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:07,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:07,560 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:07,560 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:07,560 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:07,560 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:07,560 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:07,561 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,561 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,562 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,562 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,564 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,566 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,566 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,566 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:07,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:07,566 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,566 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,566 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,566 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,566 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,566 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,566 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,566 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:07,566 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:07,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:07,628 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:56:07,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:07,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:07,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:07,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:07,629 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:07,629 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:07,666 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:07,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:07,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:07,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:07,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:07,670 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:07,670 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,670 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:07,670 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:07,671 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,671 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,673 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,673 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,674 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,674 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:07,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:07,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:07,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:07,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:07,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:07,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:07,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:07,675 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:07,676 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,676 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:07,676 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:07,676 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:07,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:07,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:07,676 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:07,679 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,679 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,679 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,679 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:07,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa3fb9a[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:07,681 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:07,681 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,682 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,683 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:07,683 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:07,683 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:07,683 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,683 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:07,684 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:07,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:07,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:07,688 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:07,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:07,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:07,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:07,688 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:07,690 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,690 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,690 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,691 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,692 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095367672, max timestamp=1681095367692Num uniques 859; 2023-04-10 02:56:07,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:07,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095367672, max timestamp=1681095367692Num uniques 859; 2023-04-10 02:56:07,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:07,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095367672, max timestamp=1681095367692Num uniques 859; 2023-04-10 02:56:07,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,696 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:07,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:07,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:07,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,696 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:07,696 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:07,696 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,735 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/2bf25f005b9e4a6cb0869c7e5b3d4f81 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2bf25f005b9e4a6cb0869c7e5b3d4f81 2023-04-10 02:56:07,735 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/7f34cca6e6e346d99280b1834ab166d6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/7f34cca6e6e346d99280b1834ab166d6 2023-04-10 02:56:07,740 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/A of 31f062e6d6eacf38c98ea51ae134e46f into 7f34cca6e6e346d99280b1834ab166d6(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:07,740 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/B of 31f062e6d6eacf38c98ea51ae134e46f into 2bf25f005b9e4a6cb0869c7e5b3d4f81(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:07,740 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:07,740 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:07,740 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/A, priority=13, startTime=1681095367312; duration=0sec 2023-04-10 02:56:07,741 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/B, priority=13, startTime=1681095367312; duration=0sec 2023-04-10 02:56:07,741 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:07,741 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:07,741 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:56:07,741 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:56:07,742 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/C is initiating minor compaction (all files) 2023-04-10 02:56:07,742 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/C in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,742 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/c9dc8a08503f4e5ebb0b627d501e074e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/faa0d787125642b7921c8d764b8d9a4f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/56ed5d05baea41e3b534afd1d7455cbb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/dc66a732b33242f58e067e5f33c2bc3a] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=48.6 K 2023-04-10 02:56:07,742 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting c9dc8a08503f4e5ebb0b627d501e074e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1681095364817 2023-04-10 02:56:07,742 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting faa0d787125642b7921c8d764b8d9a4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1681095365454 2023-04-10 02:56:07,743 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 56ed5d05baea41e3b534afd1d7455cbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681095366143 2023-04-10 02:56:07,743 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting dc66a732b33242f58e067e5f33c2bc3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=549, earliestPutTs=1681095366477 2023-04-10 02:56:07,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681095367734, max timestamp=1681095367745Num uniques 1018; 2023-04-10 02:56:07,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-10 02:56:07,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681095367734, max timestamp=1681095367745Num uniques 1018; 2023-04-10 02:56:07,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-10 02:56:07,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681095367734, max timestamp=1681095367745Num uniques 1018; 2023-04-10 02:56:07,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-10 02:56:07,751 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#C#compaction#488 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:07,763 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/f3f8237032e4490fb6d2dc0eb3e79e1e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f3f8237032e4490fb6d2dc0eb3e79e1e 2023-04-10 02:56:07,769 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/C of 31f062e6d6eacf38c98ea51ae134e46f into f3f8237032e4490fb6d2dc0eb3e79e1e(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:07,769 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:07,769 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/C, priority=12, startTime=1681095367312; duration=0sec 2023-04-10 02:56:07,769 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:07,797 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:07,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:07,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095427793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:07,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:07,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:07,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:07,800 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:07,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:07,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:07,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,800 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,801 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,801 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,801 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,801 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:07,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:07,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:07,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:07,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:07,803 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:07,803 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,803 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:07,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:07,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:07,804 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,804 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:07,804 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:07,804 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:07,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:07,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:07,804 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:07,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:07,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:07,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095427794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095427794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,806 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:07,807 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:07,808 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:07,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3e949f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:07,808 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,810 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:07,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:07,810 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:07,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,810 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:07,811 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,811 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:07,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,811 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,811 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:07,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:07,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:07,814 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:07,814 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:07,814 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:07,814 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:07,814 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:07,815 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,816 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,816 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,818 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,825 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,825 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,825 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:07,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:07,825 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,825 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,825 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,825 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:07,825 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:07,825 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,825 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,825 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,825 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:07,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56200 deadline: 1681095427885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,893 DEBUG [Thread-2981] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:56:07,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:07,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56216 deadline: 1681095427890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,896 DEBUG [Thread-2983] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:56:07,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:07,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095427898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:07,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095427905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:07,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095427906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,925 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:07,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:07,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:07,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:07,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:07,928 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:07,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:07,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:07,929 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,929 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,930 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,930 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,930 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,930 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:07,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:07,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:07,931 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:07,931 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:07,931 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:07,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:07,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:07,932 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,932 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:07,932 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:07,932 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:07,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:07,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:07,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:07,932 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:07,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,934 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,934 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,934 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:07,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e5131c3[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:07,935 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:07,935 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,936 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,937 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:07,937 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:07,937 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:07,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,937 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:07,939 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,939 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:07,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,939 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,939 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:07,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:07,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:07,941 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:07,941 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:07,941 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:07,941 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:07,941 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:07,943 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,943 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:07,943 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:07,943 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,945 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:07,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:07,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:07,948 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:07,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:07,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:07,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:07,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:07,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:07,948 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:07,948 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:07,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:07,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:07,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:07,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:08,048 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:08,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:08,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:08,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:08,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:08,051 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:08,051 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:08,051 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:08,051 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:08,052 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,052 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,053 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,053 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:08,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:08,053 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,053 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:08,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:08,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:08,054 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:08,054 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:08,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:08,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:08,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:08,055 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:08,055 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:08,055 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:08,055 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:08,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:08,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:08,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:08,055 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:08,058 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,058 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:08,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:08,059 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,059 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:08,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:08,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57755b96[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:08,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:08,060 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,060 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/1bd2f19ebfb842a5b9f98a0f626db9b4 2023-04-10 02:56:08,061 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:08,062 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,062 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:08,063 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:08,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:08,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:08,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:08,065 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:08,065 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:08,065 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:08,065 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:08,066 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:08,068 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,068 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:08,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:08,068 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:08,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:08,069 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:08,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:08,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:08,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,070 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,074 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:08,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:08,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:08,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:08,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:08,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:08,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:08,075 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:08,075 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:08,074 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,075 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,075 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:08,075 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,075 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:08,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095428106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:08,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095428108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:08,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095428109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,175 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:08,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:08,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:08,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:08,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:08,178 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:08,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:08,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:08,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:08,180 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,180 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,181 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:08,181 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:08,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:08,181 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,182 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:08,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:08,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:08,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:08,182 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:08,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:08,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:08,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:08,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:08,183 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:08,183 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:08,183 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:08,183 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:08,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:08,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:08,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:08,183 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:08,185 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,185 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:08,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:08,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:08,195 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,195 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:08,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:08,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:08,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27fa05c9[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:08,199 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:08,199 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,200 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:08,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:08,200 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,200 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:08,200 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:08,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:08,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:08,202 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:08,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:08,206 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:08,206 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:08,206 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:08,206 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:08,206 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:08,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:08,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,208 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:08,210 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:08,213 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,213 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,213 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:08,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:08,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:08,213 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:08,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:08,213 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:08,213 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:08,213 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:08,213 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,213 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,213 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,213 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,213 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:08,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:08,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:08,313 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:08,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:08,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:08,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:08,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:08,317 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:08,317 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:08,317 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:08,317 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:08,317 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,317 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,325 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:08,325 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:08,326 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,326 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:08,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:08,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:08,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:08,327 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:08,328 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:08,328 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:08,328 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:08,330 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f58e7c5[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:08,332 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:08,332 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,333 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:08,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:08,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:08,334 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,334 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:08,335 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:08,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:08,338 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:08,338 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:08,338 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:08,338 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:08,338 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:08,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:08,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:08,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:08,341 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,342 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,344 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:08,344 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:08,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:08,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:08,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:08,344 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,345 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:08,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:08,344 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:08,345 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:08,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:08,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:08,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56260 deadline: 1681095428415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:08,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56256 deadline: 1681095428416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:08,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56198 deadline: 1681095428417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,445 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:08,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:08,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:08,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:08,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:08,448 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:08,448 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:08,448 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:08,448 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:08,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,450 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,450 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:08,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:08,450 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,450 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:08,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:08,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:08,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:08,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:08,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:08,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:08,452 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:08,452 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:08,452 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:08,452 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:08,452 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:08,452 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:08,452 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:08,452 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:08,452 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,452 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:08,452 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:08,454 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,454 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:08,454 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,454 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:08,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@382002c4[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:08,456 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:08,456 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,457 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,458 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:08,458 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:08,458 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:08,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:08,461 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,461 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:08,461 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:08,461 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:08,463 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:08,463 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:08,463 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:08,463 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:08,463 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:08,465 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,465 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:08,465 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,466 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:08,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:08,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:08,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:08,467 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:08,469 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:08,469 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,469 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,469 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,469 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,469 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,469 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:08,469 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:08,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:08,469 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:08,469 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:08,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:08,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:08,470 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:08,471 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:08,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:08,473 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/77fd45239a2d4401bfe999449725045e 2023-04-10 02:56:08,484 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/41d19752940c49a38685bcd343727be1 2023-04-10 02:56:08,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/1bd2f19ebfb842a5b9f98a0f626db9b4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/1bd2f19ebfb842a5b9f98a0f626db9b4 2023-04-10 02:56:08,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/1bd2f19ebfb842a5b9f98a0f626db9b4, entries=100, sequenceid=560, filesize=9.5 K 2023-04-10 02:56:08,499 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/77fd45239a2d4401bfe999449725045e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/77fd45239a2d4401bfe999449725045e 2023-04-10 02:56:08,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/77fd45239a2d4401bfe999449725045e, entries=100, sequenceid=560, filesize=9.5 K 2023-04-10 02:56:08,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/41d19752940c49a38685bcd343727be1 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/41d19752940c49a38685bcd343727be1 2023-04-10 02:56:08,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/41d19752940c49a38685bcd343727be1, entries=100, sequenceid=560, filesize=9.5 K 2023-04-10 02:56:08,506 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 31f062e6d6eacf38c98ea51ae134e46f in 878ms, sequenceid=560, compaction requested=true 2023-04-10 02:56:08,506 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:08,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:08,506 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:08,506 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:08,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:08,507 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:08,507 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:08,507 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/A is initiating minor compaction (all files) 2023-04-10 02:56:08,507 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/A in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:08,507 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/7f34cca6e6e346d99280b1834ab166d6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/022c174ef35a413d98727356a599fc35, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/1bd2f19ebfb842a5b9f98a0f626db9b4] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=34.4 K 2023-04-10 02:56:08,508 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:08,508 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 31f062e6d6eacf38c98ea51ae134e46f/B is initiating minor compaction (all files) 2023-04-10 02:56:08,508 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 31f062e6d6eacf38c98ea51ae134e46f/B in TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:08,508 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2bf25f005b9e4a6cb0869c7e5b3d4f81, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/9a43baae2fd4473f9ee4a1f1d49e506c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/77fd45239a2d4401bfe999449725045e] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp, totalSize=34.4 K 2023-04-10 02:56:08,508 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 7f34cca6e6e346d99280b1834ab166d6, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681095366143 2023-04-10 02:56:08,508 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 022c174ef35a413d98727356a599fc35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=549, earliestPutTs=1681095366477 2023-04-10 02:56:08,508 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 2bf25f005b9e4a6cb0869c7e5b3d4f81, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681095366143 2023-04-10 02:56:08,509 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 1bd2f19ebfb842a5b9f98a0f626db9b4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1681095367623 2023-04-10 02:56:08,509 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 9a43baae2fd4473f9ee4a1f1d49e506c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=549, earliestPutTs=1681095366477 2023-04-10 02:56:08,510 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 77fd45239a2d4401bfe999449725045e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1681095367623 2023-04-10 02:56:08,515 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#A#compaction#491 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:08,520 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 31f062e6d6eacf38c98ea51ae134e46f#B#compaction#492 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:08,529 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/506c1778f2644862864a1d391c1b15d3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/506c1778f2644862864a1d391c1b15d3 2023-04-10 02:56:08,533 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/B of 31f062e6d6eacf38c98ea51ae134e46f into 506c1778f2644862864a1d391c1b15d3(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:08,533 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:08,533 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/B, priority=13, startTime=1681095368506; duration=0sec 2023-04-10 02:56:08,533 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:08,534 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-10 02:56:08,534 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-10 02:56:08,534 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-10 02:56:08,534 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. because compaction request was cancelled 2023-04-10 02:56:08,571 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:08,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:08,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:08,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:08,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:08,575 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:08,575 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:08,575 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:08,575 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:08,575 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,575 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,578 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:08,578 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:08,578 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:08,578 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:08,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:08,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:08,579 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:08,579 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. started... 2023-04-10 02:56:08,579 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-10 02:56:08,579 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:08,684 DEBUG [Thread-2986] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x211582f2 to 127.0.0.1:60568 2023-04-10 02:56:08,684 DEBUG [Thread-2986] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:08,685 DEBUG [Thread-2994] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3011deb3 to 127.0.0.1:60568 2023-04-10 02:56:08,686 DEBUG [Thread-2994] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:08,688 DEBUG [Thread-2988] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65816b6e to 127.0.0.1:60568 2023-04-10 02:56:08,688 DEBUG [Thread-2988] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:08,689 DEBUG [Thread-2990] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x653bb8fc to 127.0.0.1:60568 2023-04-10 02:56:08,689 DEBUG [Thread-2990] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:08,690 DEBUG [Thread-2992] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x720d29f5 to 127.0.0.1:60568 2023-04-10 02:56:08,690 DEBUG [Thread-2992] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:08,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:08,921 DEBUG [Thread-2979] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fb168ce to 127.0.0.1:60568 2023-04-10 02:56:08,921 DEBUG [Thread-2979] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:08,921 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. as already flushing 2023-04-10 02:56:08,922 DEBUG [Thread-2977] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06edd017 to 127.0.0.1:60568 2023-04-10 02:56:08,922 DEBUG [Thread-2977] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:08,924 DEBUG [Thread-2975] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c29757d to 127.0.0.1:60568 2023-04-10 02:56:08,924 DEBUG [Thread-2975] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:08,927 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/c1ffb93df2174d468d0ebbeaf509234f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/c1ffb93df2174d468d0ebbeaf509234f 2023-04-10 02:56:08,931 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 31f062e6d6eacf38c98ea51ae134e46f/A of 31f062e6d6eacf38c98ea51ae134e46f into c1ffb93df2174d468d0ebbeaf509234f(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:08,931 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:08,931 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f., storeName=31f062e6d6eacf38c98ea51ae134e46f/A, priority=13, startTime=1681095368506; duration=0sec 2023-04-10 02:56:08,932 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:08,986 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/03b718a8f7ab4427a0433def85ed2b82 2023-04-10 02:56:09,397 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/afd5a2a653bd40cda2d6f2b8d6c16d37 2023-04-10 02:56:09,807 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/213547a6db3c4dcfa87fe94afa761fda 2023-04-10 02:56:09,810 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/03b718a8f7ab4427a0433def85ed2b82 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/03b718a8f7ab4427a0433def85ed2b82 2023-04-10 02:56:09,812 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/03b718a8f7ab4427a0433def85ed2b82, entries=150, sequenceid=617, filesize=11.9 K 2023-04-10 02:56:09,813 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/afd5a2a653bd40cda2d6f2b8d6c16d37 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/afd5a2a653bd40cda2d6f2b8d6c16d37 2023-04-10 02:56:09,816 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/afd5a2a653bd40cda2d6f2b8d6c16d37, entries=150, sequenceid=617, filesize=11.9 K 2023-04-10 02:56:09,816 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/213547a6db3c4dcfa87fe94afa761fda as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/213547a6db3c4dcfa87fe94afa761fda 2023-04-10 02:56:09,819 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/213547a6db3c4dcfa87fe94afa761fda, entries=150, sequenceid=617, filesize=11.9 K 2023-04-10 02:56:09,820 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=20.13 KB/20610 for 31f062e6d6eacf38c98ea51ae134e46f in 1241ms, sequenceid=617, compaction requested=true 2023-04-10 02:56:09,820 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:09,820 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:09,820 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:09,820 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:09,820 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,820 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:09,820 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:09,821 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:09,821 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@457447bd[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:09,823 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:09,823 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:09,824 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:09,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:09,824 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:09,824 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:09,824 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,824 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:09,824 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:09,825 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:09,825 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:09,825 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,825 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:09,827 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:09,827 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:09,827 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:09,827 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:09,827 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:09,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:09,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:09,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:09,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:09,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:09,830 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:09,831 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:09,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:09,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:09,833 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:09,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:09,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:09,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:09,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:09,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:09,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:09,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:09,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:09,833 DEBUG [Thread-2985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:09,833 DEBUG [Thread-2985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:09,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:09,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:09,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:09,933 DEBUG [Thread-2985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:09,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:11,900 DEBUG [Thread-2983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63093e21 to 127.0.0.1:60568 2023-04-10 02:56:11,900 DEBUG [Thread-2983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:11,917 DEBUG [Thread-2981] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x461f5f3f to 127.0.0.1:60568 2023-04-10 02:56:11,917 DEBUG [Thread-2981] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 114 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 121 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 123 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 73 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 69 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 1908 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 5724 rows 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 1903 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 5709 rows 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 1903 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 5707 rows 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 1897 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 5691 rows 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(402): scanned 1901 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(403): verified 5703 rows 2023-04-10 02:56:11,918 INFO [Listener at localhost/44533] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-10 02:56:11,918 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34ad4aa8 to 127.0.0.1:60568 2023-04-10 02:56:11,919 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:11,920 INFO [Listener at localhost/44533] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-10 02:56:11,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-10 02:56:11,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:11,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-10 02:56:11,924 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095371924"}]},"ts":"1681095371924"} 2023-04-10 02:56:11,925 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-10 02:56:11,927 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-10 02:56:11,927 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=31f062e6d6eacf38c98ea51ae134e46f, UNASSIGN}] 2023-04-10 02:56:11,929 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=31f062e6d6eacf38c98ea51ae134e46f, UNASSIGN 2023-04-10 02:56:11,929 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=31f062e6d6eacf38c98ea51ae134e46f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:11,929 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095371929"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095371929"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095371929"}]},"ts":"1681095371929"} 2023-04-10 02:56:11,930 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:56:12,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-10 02:56:12,082 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:12,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 31f062e6d6eacf38c98ea51ae134e46f, disabling compactions & flushes 2023-04-10 02:56:12,082 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:12,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:12,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. after waiting 0 ms 2023-04-10 02:56:12,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:12,082 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 31f062e6d6eacf38c98ea51ae134e46f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-10 02:56:12,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=A 2023-04-10 02:56:12,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:12,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=B 2023-04-10 02:56:12,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:12,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 31f062e6d6eacf38c98ea51ae134e46f, store=C 2023-04-10 02:56:12,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:12,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-10 02:56:12,488 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/3f509059fa774264a2ecfea4410f47a6 2023-04-10 02:56:12,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-10 02:56:12,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/e78d96c3c70247c69c6976e92accb0ab 2023-04-10 02:56:13,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-10 02:56:13,304 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/4ac7405b084341f38ef1e920f693b38f 2023-04-10 02:56:13,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/A/3f509059fa774264a2ecfea4410f47a6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/3f509059fa774264a2ecfea4410f47a6 2023-04-10 02:56:13,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/3f509059fa774264a2ecfea4410f47a6, entries=150, sequenceid=626, filesize=11.9 K 2023-04-10 02:56:13,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/B/e78d96c3c70247c69c6976e92accb0ab as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/e78d96c3c70247c69c6976e92accb0ab 2023-04-10 02:56:13,313 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/e78d96c3c70247c69c6976e92accb0ab, entries=150, sequenceid=626, filesize=11.9 K 2023-04-10 02:56:13,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/.tmp/C/4ac7405b084341f38ef1e920f693b38f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/4ac7405b084341f38ef1e920f693b38f 2023-04-10 02:56:13,316 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/4ac7405b084341f38ef1e920f693b38f, entries=150, sequenceid=626, filesize=11.9 K 2023-04-10 02:56:13,317 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 31f062e6d6eacf38c98ea51ae134e46f in 1235ms, sequenceid=626, compaction requested=true 2023-04-10 02:56:13,341 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/1f80bca6608440b7a4a086832c18e4eb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/fd4734768ed64813b7c4ac1d7613a276, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ba0255fef08e4f1f9cf008204b70ad51, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/b956828c407f46cfaab8b10d38f711f1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/861a51d7eefa475b9d81747d4f35b64d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/fd3d9bd2555d4ff298c2109ee75efb98, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/dff2d463a1ef40a28ed2b37cf5088a11, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/83cecfba865e496cb6a214b423730817, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ba32a4b60c8c4d9389d3f11f05ebb3fd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/331abe4fd04244cda36c014ad7f478ac, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ae4c5c3715bf4326b999602cb19fd1b1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/3c785b1a1e5246d08bbf62b8616704b3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/4091c4885357409db57927372ee54901, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/8aea0de88c9a4c81b6b133afaa1b07b2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/0a9ba0b37d05411daa5d58a3a2b7021e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/5eb7149ca1d84d29a4fc569f4153f275, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/018fd21d6b8444a3b2438e266541eb7c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/6974b42651c64cbab0eb2b75b201ba86, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/9a1b04e14a6046f9a5960ed19421bcc5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/06f4f84c19d64d2b9407a1ed0326d707, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/8478bfeff2bb4931be02ed3d10752fd8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/058a3783cc774c4798f03ca74d2bf0fa, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/010e27a62b87462c8b16fcf6791ba52d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/75aa1dd627714233bd3b8f9e4d0462a6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ea1a82cbd9224c179a84fed8b1debe13, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/36c9004b217b497782dcae22f9f83620, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/e7c34ec9e255401bb12c77efdafe9743, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/0052d694695f4133b4de53e45dde6d5a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/e8360a4beebd433f81eeb97b21d4db65, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/f0667e9d92984672bddb84c8d3373ec2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/a7ccd70c93a745588296c6efd8f9e6a3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/f93da79f667a4e109baebd7c4b7ec6cc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/a7eb6b4c3e5b4f0baa89fd348a351dc5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/7f34cca6e6e346d99280b1834ab166d6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/022c174ef35a413d98727356a599fc35, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/1bd2f19ebfb842a5b9f98a0f626db9b4] to archive 2023-04-10 02:56:13,342 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:56:13,344 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/1f80bca6608440b7a4a086832c18e4eb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/1f80bca6608440b7a4a086832c18e4eb 2023-04-10 02:56:13,345 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/fd4734768ed64813b7c4ac1d7613a276 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/fd4734768ed64813b7c4ac1d7613a276 2023-04-10 02:56:13,345 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ba0255fef08e4f1f9cf008204b70ad51 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ba0255fef08e4f1f9cf008204b70ad51 2023-04-10 02:56:13,346 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/b956828c407f46cfaab8b10d38f711f1 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/b956828c407f46cfaab8b10d38f711f1 2023-04-10 02:56:13,347 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/861a51d7eefa475b9d81747d4f35b64d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/861a51d7eefa475b9d81747d4f35b64d 2023-04-10 02:56:13,347 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/fd3d9bd2555d4ff298c2109ee75efb98 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/fd3d9bd2555d4ff298c2109ee75efb98 2023-04-10 02:56:13,348 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/dff2d463a1ef40a28ed2b37cf5088a11 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/dff2d463a1ef40a28ed2b37cf5088a11 2023-04-10 02:56:13,349 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/83cecfba865e496cb6a214b423730817 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/83cecfba865e496cb6a214b423730817 2023-04-10 02:56:13,350 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ba32a4b60c8c4d9389d3f11f05ebb3fd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ba32a4b60c8c4d9389d3f11f05ebb3fd 2023-04-10 02:56:13,350 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/331abe4fd04244cda36c014ad7f478ac to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/331abe4fd04244cda36c014ad7f478ac 2023-04-10 02:56:13,351 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ae4c5c3715bf4326b999602cb19fd1b1 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ae4c5c3715bf4326b999602cb19fd1b1 2023-04-10 02:56:13,352 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/3c785b1a1e5246d08bbf62b8616704b3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/3c785b1a1e5246d08bbf62b8616704b3 2023-04-10 02:56:13,353 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/4091c4885357409db57927372ee54901 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/4091c4885357409db57927372ee54901 2023-04-10 02:56:13,353 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/8aea0de88c9a4c81b6b133afaa1b07b2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/8aea0de88c9a4c81b6b133afaa1b07b2 2023-04-10 02:56:13,354 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/0a9ba0b37d05411daa5d58a3a2b7021e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/0a9ba0b37d05411daa5d58a3a2b7021e 2023-04-10 02:56:13,355 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/5eb7149ca1d84d29a4fc569f4153f275 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/5eb7149ca1d84d29a4fc569f4153f275 2023-04-10 02:56:13,356 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/018fd21d6b8444a3b2438e266541eb7c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/018fd21d6b8444a3b2438e266541eb7c 2023-04-10 02:56:13,356 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/6974b42651c64cbab0eb2b75b201ba86 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/6974b42651c64cbab0eb2b75b201ba86 2023-04-10 02:56:13,357 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/9a1b04e14a6046f9a5960ed19421bcc5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/9a1b04e14a6046f9a5960ed19421bcc5 2023-04-10 02:56:13,358 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/06f4f84c19d64d2b9407a1ed0326d707 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/06f4f84c19d64d2b9407a1ed0326d707 2023-04-10 02:56:13,359 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/8478bfeff2bb4931be02ed3d10752fd8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/8478bfeff2bb4931be02ed3d10752fd8 2023-04-10 02:56:13,359 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/058a3783cc774c4798f03ca74d2bf0fa to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/058a3783cc774c4798f03ca74d2bf0fa 2023-04-10 02:56:13,360 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/010e27a62b87462c8b16fcf6791ba52d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/010e27a62b87462c8b16fcf6791ba52d 2023-04-10 02:56:13,361 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/75aa1dd627714233bd3b8f9e4d0462a6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/75aa1dd627714233bd3b8f9e4d0462a6 2023-04-10 02:56:13,362 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ea1a82cbd9224c179a84fed8b1debe13 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/ea1a82cbd9224c179a84fed8b1debe13 2023-04-10 02:56:13,362 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/36c9004b217b497782dcae22f9f83620 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/36c9004b217b497782dcae22f9f83620 2023-04-10 02:56:13,363 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/e7c34ec9e255401bb12c77efdafe9743 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/e7c34ec9e255401bb12c77efdafe9743 2023-04-10 02:56:13,364 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/0052d694695f4133b4de53e45dde6d5a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/0052d694695f4133b4de53e45dde6d5a 2023-04-10 02:56:13,365 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/e8360a4beebd433f81eeb97b21d4db65 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/e8360a4beebd433f81eeb97b21d4db65 2023-04-10 02:56:13,366 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/f0667e9d92984672bddb84c8d3373ec2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/f0667e9d92984672bddb84c8d3373ec2 2023-04-10 02:56:13,367 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/a7ccd70c93a745588296c6efd8f9e6a3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/a7ccd70c93a745588296c6efd8f9e6a3 2023-04-10 02:56:13,368 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/f93da79f667a4e109baebd7c4b7ec6cc to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/f93da79f667a4e109baebd7c4b7ec6cc 2023-04-10 02:56:13,368 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/a7eb6b4c3e5b4f0baa89fd348a351dc5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/a7eb6b4c3e5b4f0baa89fd348a351dc5 2023-04-10 02:56:13,369 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/7f34cca6e6e346d99280b1834ab166d6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/7f34cca6e6e346d99280b1834ab166d6 2023-04-10 02:56:13,370 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/022c174ef35a413d98727356a599fc35 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/022c174ef35a413d98727356a599fc35 2023-04-10 02:56:13,371 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/1bd2f19ebfb842a5b9f98a0f626db9b4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/1bd2f19ebfb842a5b9f98a0f626db9b4 2023-04-10 02:56:13,374 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d49dfb784e2745599090498385cbc685, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2a544e8b22284de3a642144c42e34a3d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/6513ea61fb7b49799d43c9a50294e731, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/894738f7084a40079a270a31fabece46, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/fbf7977a10a247bd8dc9c19a05d70fa6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/bd730a5648d34241963bb01b2b3d0d31, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/195f341bd07d4db3b9d210e73f28e5b8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/ceccbe5c8c8244eab0b494e8a88639bd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/94b091a6a2d34ce8a979acd2ea8af2c9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/1dd19299ae0846fbaec3559bf8523f7a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/cc9abc9abbc64fb08650ee1de48bb2b0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/c57a79a9e9f64796a124ef4be968a5aa, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d600ef5ee6664763807f307d2e26284c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/f34f066e73224342ae90bb0d38a03ad6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/9fd7e9be720748f2b41bb65f3ab68ca9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/0f4e3a6dcacd4e28943f2e906b9887e4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/e9d50c9cf9bc40c6898546030820628a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/793d13a988844576a40398f6441e232d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/f8fa91016fd54112b02f2152afbd5f36, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7e32d3deddd04639b5c92b93b669709b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/99e8e740bdd041e09f14969ca41b4389, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/5645a46c898947f9aca252deed0dbbfe, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/36531af925c64dcb8aec0dd619fbfe87, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/29b8287c9e464269bf4999655beff4d0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/c7272ae278c24ceab0630cdc3de2710b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7132a2807b83490fa8aaee39d23dd4cc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2d63a2c2a8ef446890cd232c2ad9c951, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7db9897eef74447d921bd4dc5dec3147, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/45ef2d90e6c3456cbb151cad2a852d74, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/efeded2691b94991969c9105b06a6b35, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d8631e4bdfd54e73bc10e8cc437b1342, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/de47b335bdcc4526a89f772c5933adcc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2bf25f005b9e4a6cb0869c7e5b3d4f81, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/cec03427b33a48028cd983eb1e281fb0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/9a43baae2fd4473f9ee4a1f1d49e506c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/77fd45239a2d4401bfe999449725045e] to archive 2023-04-10 02:56:13,374 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:56:13,375 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d49dfb784e2745599090498385cbc685 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d49dfb784e2745599090498385cbc685 2023-04-10 02:56:13,376 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2a544e8b22284de3a642144c42e34a3d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2a544e8b22284de3a642144c42e34a3d 2023-04-10 02:56:13,377 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/6513ea61fb7b49799d43c9a50294e731 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/6513ea61fb7b49799d43c9a50294e731 2023-04-10 02:56:13,378 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/894738f7084a40079a270a31fabece46 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/894738f7084a40079a270a31fabece46 2023-04-10 02:56:13,379 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/fbf7977a10a247bd8dc9c19a05d70fa6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/fbf7977a10a247bd8dc9c19a05d70fa6 2023-04-10 02:56:13,380 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/bd730a5648d34241963bb01b2b3d0d31 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/bd730a5648d34241963bb01b2b3d0d31 2023-04-10 02:56:13,380 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/195f341bd07d4db3b9d210e73f28e5b8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/195f341bd07d4db3b9d210e73f28e5b8 2023-04-10 02:56:13,381 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/ceccbe5c8c8244eab0b494e8a88639bd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/ceccbe5c8c8244eab0b494e8a88639bd 2023-04-10 02:56:13,382 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/94b091a6a2d34ce8a979acd2ea8af2c9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/94b091a6a2d34ce8a979acd2ea8af2c9 2023-04-10 02:56:13,383 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/1dd19299ae0846fbaec3559bf8523f7a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/1dd19299ae0846fbaec3559bf8523f7a 2023-04-10 02:56:13,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/cc9abc9abbc64fb08650ee1de48bb2b0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/cc9abc9abbc64fb08650ee1de48bb2b0 2023-04-10 02:56:13,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/c57a79a9e9f64796a124ef4be968a5aa to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/c57a79a9e9f64796a124ef4be968a5aa 2023-04-10 02:56:13,385 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d600ef5ee6664763807f307d2e26284c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d600ef5ee6664763807f307d2e26284c 2023-04-10 02:56:13,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/f34f066e73224342ae90bb0d38a03ad6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/f34f066e73224342ae90bb0d38a03ad6 2023-04-10 02:56:13,387 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/9fd7e9be720748f2b41bb65f3ab68ca9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/9fd7e9be720748f2b41bb65f3ab68ca9 2023-04-10 02:56:13,387 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/0f4e3a6dcacd4e28943f2e906b9887e4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/0f4e3a6dcacd4e28943f2e906b9887e4 2023-04-10 02:56:13,388 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/e9d50c9cf9bc40c6898546030820628a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/e9d50c9cf9bc40c6898546030820628a 2023-04-10 02:56:13,389 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/793d13a988844576a40398f6441e232d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/793d13a988844576a40398f6441e232d 2023-04-10 02:56:13,390 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/f8fa91016fd54112b02f2152afbd5f36 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/f8fa91016fd54112b02f2152afbd5f36 2023-04-10 02:56:13,390 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7e32d3deddd04639b5c92b93b669709b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7e32d3deddd04639b5c92b93b669709b 2023-04-10 02:56:13,391 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/99e8e740bdd041e09f14969ca41b4389 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/99e8e740bdd041e09f14969ca41b4389 2023-04-10 02:56:13,392 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/5645a46c898947f9aca252deed0dbbfe to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/5645a46c898947f9aca252deed0dbbfe 2023-04-10 02:56:13,393 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/36531af925c64dcb8aec0dd619fbfe87 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/36531af925c64dcb8aec0dd619fbfe87 2023-04-10 02:56:13,393 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/29b8287c9e464269bf4999655beff4d0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/29b8287c9e464269bf4999655beff4d0 2023-04-10 02:56:13,394 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/c7272ae278c24ceab0630cdc3de2710b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/c7272ae278c24ceab0630cdc3de2710b 2023-04-10 02:56:13,395 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7132a2807b83490fa8aaee39d23dd4cc to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7132a2807b83490fa8aaee39d23dd4cc 2023-04-10 02:56:13,396 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2d63a2c2a8ef446890cd232c2ad9c951 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2d63a2c2a8ef446890cd232c2ad9c951 2023-04-10 02:56:13,397 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7db9897eef74447d921bd4dc5dec3147 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/7db9897eef74447d921bd4dc5dec3147 2023-04-10 02:56:13,397 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/45ef2d90e6c3456cbb151cad2a852d74 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/45ef2d90e6c3456cbb151cad2a852d74 2023-04-10 02:56:13,398 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/efeded2691b94991969c9105b06a6b35 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/efeded2691b94991969c9105b06a6b35 2023-04-10 02:56:13,399 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d8631e4bdfd54e73bc10e8cc437b1342 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/d8631e4bdfd54e73bc10e8cc437b1342 2023-04-10 02:56:13,399 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/de47b335bdcc4526a89f772c5933adcc to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/de47b335bdcc4526a89f772c5933adcc 2023-04-10 02:56:13,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2bf25f005b9e4a6cb0869c7e5b3d4f81 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/2bf25f005b9e4a6cb0869c7e5b3d4f81 2023-04-10 02:56:13,401 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/cec03427b33a48028cd983eb1e281fb0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/cec03427b33a48028cd983eb1e281fb0 2023-04-10 02:56:13,401 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/9a43baae2fd4473f9ee4a1f1d49e506c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/9a43baae2fd4473f9ee4a1f1d49e506c 2023-04-10 02:56:13,402 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/77fd45239a2d4401bfe999449725045e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/77fd45239a2d4401bfe999449725045e 2023-04-10 02:56:13,405 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/4efa26ea214d4ce3b75a87ea146f355d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/517aca225b7c40c99ba6bf44d25262ba, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/633b95d5d0194217b324e0c497f60059, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/b1fcaaf359eb419cae28e15e6b44afb5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/0633939e6bd94b19ad4d29a34b4b4764, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/2f52354cb7444ca08e493572b555a856, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/d351f3b016e7493484712b06121a3bb3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f705569c77ea4b708d261351201a4a68, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/d7200dd177b74b569ea24465488e56fe, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f0d57df5829842ed96047c2c0487691c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/076be910bc274927b143b022d268e758, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/a0056bc2463d43af8b42e31088fd7efe, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/a10944042575431ca39379f68e6252c2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/847acd23978f4776aa04f3e1b29a3c12, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/5e3e7ee22c5a4ba6b16d8ea05c3b9dd8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/12199aa8920346b7b7585bd2da60c155, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/afe2c25a759940aeae3541de719c0cb2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/3d189fcc544940719fc1142446042568, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/bf1b565a73d0445d8271488286d23c6d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/db9941ec72b34bcfa5e6afcabdbffa82, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/537e26d840e948538b7ae7d47ad5e82d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/b2f671d8fc83485c8a67f00c7b9b4365, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/efc43354fba74453babea5c9a887b483, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/42a9b0b992a34cb2a529024409b4bfb8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/ec421356b2b64971a9d0fd558a3ccc21, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/8a2254faeb60469585fdd4cc7d62cdc7, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/ea7ddf745230464e8c5932c998879ddd, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/e364c419efaf48f0823479a2ad8c0e67, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/6c8b8518f1c5427a9f1e16e92c55758d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/c9dc8a08503f4e5ebb0b627d501e074e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/7513ae5f48f641baa6eac93cf52a142d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/faa0d787125642b7921c8d764b8d9a4f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/56ed5d05baea41e3b534afd1d7455cbb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/dc66a732b33242f58e067e5f33c2bc3a] to archive 2023-04-10 02:56:13,406 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:56:13,407 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/4efa26ea214d4ce3b75a87ea146f355d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/4efa26ea214d4ce3b75a87ea146f355d 2023-04-10 02:56:13,408 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/517aca225b7c40c99ba6bf44d25262ba to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/517aca225b7c40c99ba6bf44d25262ba 2023-04-10 02:56:13,409 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/633b95d5d0194217b324e0c497f60059 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/633b95d5d0194217b324e0c497f60059 2023-04-10 02:56:13,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/b1fcaaf359eb419cae28e15e6b44afb5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/b1fcaaf359eb419cae28e15e6b44afb5 2023-04-10 02:56:13,411 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/0633939e6bd94b19ad4d29a34b4b4764 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/0633939e6bd94b19ad4d29a34b4b4764 2023-04-10 02:56:13,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/2f52354cb7444ca08e493572b555a856 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/2f52354cb7444ca08e493572b555a856 2023-04-10 02:56:13,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/d351f3b016e7493484712b06121a3bb3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/d351f3b016e7493484712b06121a3bb3 2023-04-10 02:56:13,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f705569c77ea4b708d261351201a4a68 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f705569c77ea4b708d261351201a4a68 2023-04-10 02:56:13,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/d7200dd177b74b569ea24465488e56fe to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/d7200dd177b74b569ea24465488e56fe 2023-04-10 02:56:13,416 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f0d57df5829842ed96047c2c0487691c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f0d57df5829842ed96047c2c0487691c 2023-04-10 02:56:13,417 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/076be910bc274927b143b022d268e758 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/076be910bc274927b143b022d268e758 2023-04-10 02:56:13,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/a0056bc2463d43af8b42e31088fd7efe to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/a0056bc2463d43af8b42e31088fd7efe 2023-04-10 02:56:13,419 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/a10944042575431ca39379f68e6252c2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/a10944042575431ca39379f68e6252c2 2023-04-10 02:56:13,420 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/847acd23978f4776aa04f3e1b29a3c12 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/847acd23978f4776aa04f3e1b29a3c12 2023-04-10 02:56:13,421 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/5e3e7ee22c5a4ba6b16d8ea05c3b9dd8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/5e3e7ee22c5a4ba6b16d8ea05c3b9dd8 2023-04-10 02:56:13,422 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/12199aa8920346b7b7585bd2da60c155 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/12199aa8920346b7b7585bd2da60c155 2023-04-10 02:56:13,423 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/afe2c25a759940aeae3541de719c0cb2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/afe2c25a759940aeae3541de719c0cb2 2023-04-10 02:56:13,423 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/3d189fcc544940719fc1142446042568 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/3d189fcc544940719fc1142446042568 2023-04-10 02:56:13,424 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/bf1b565a73d0445d8271488286d23c6d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/bf1b565a73d0445d8271488286d23c6d 2023-04-10 02:56:13,425 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/db9941ec72b34bcfa5e6afcabdbffa82 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/db9941ec72b34bcfa5e6afcabdbffa82 2023-04-10 02:56:13,426 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/537e26d840e948538b7ae7d47ad5e82d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/537e26d840e948538b7ae7d47ad5e82d 2023-04-10 02:56:13,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/b2f671d8fc83485c8a67f00c7b9b4365 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/b2f671d8fc83485c8a67f00c7b9b4365 2023-04-10 02:56:13,429 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/efc43354fba74453babea5c9a887b483 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/efc43354fba74453babea5c9a887b483 2023-04-10 02:56:13,429 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/42a9b0b992a34cb2a529024409b4bfb8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/42a9b0b992a34cb2a529024409b4bfb8 2023-04-10 02:56:13,430 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/ec421356b2b64971a9d0fd558a3ccc21 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/ec421356b2b64971a9d0fd558a3ccc21 2023-04-10 02:56:13,431 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/8a2254faeb60469585fdd4cc7d62cdc7 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/8a2254faeb60469585fdd4cc7d62cdc7 2023-04-10 02:56:13,432 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/ea7ddf745230464e8c5932c998879ddd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/ea7ddf745230464e8c5932c998879ddd 2023-04-10 02:56:13,432 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/e364c419efaf48f0823479a2ad8c0e67 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/e364c419efaf48f0823479a2ad8c0e67 2023-04-10 02:56:13,433 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/6c8b8518f1c5427a9f1e16e92c55758d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/6c8b8518f1c5427a9f1e16e92c55758d 2023-04-10 02:56:13,434 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/c9dc8a08503f4e5ebb0b627d501e074e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/c9dc8a08503f4e5ebb0b627d501e074e 2023-04-10 02:56:13,435 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/7513ae5f48f641baa6eac93cf52a142d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/7513ae5f48f641baa6eac93cf52a142d 2023-04-10 02:56:13,436 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/faa0d787125642b7921c8d764b8d9a4f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/faa0d787125642b7921c8d764b8d9a4f 2023-04-10 02:56:13,437 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/56ed5d05baea41e3b534afd1d7455cbb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/56ed5d05baea41e3b534afd1d7455cbb 2023-04-10 02:56:13,438 DEBUG [StoreCloser-TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/dc66a732b33242f58e067e5f33c2bc3a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/dc66a732b33242f58e067e5f33c2bc3a 2023-04-10 02:56:13,441 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/recovered.edits/629.seqid, newMaxSeqId=629, maxSeqId=1 2023-04-10 02:56:13,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f. 2023-04-10 02:56:13,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 31f062e6d6eacf38c98ea51ae134e46f: 2023-04-10 02:56:13,444 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:13,444 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=31f062e6d6eacf38c98ea51ae134e46f, regionState=CLOSED 2023-04-10 02:56:13,444 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095373444"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095373444"}]},"ts":"1681095373444"} 2023-04-10 02:56:13,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-10 02:56:13,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 31f062e6d6eacf38c98ea51ae134e46f, server=jenkins-hbase4.apache.org,38321,1681095222501 in 1.5150 sec 2023-04-10 02:56:13,447 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-10 02:56:13,447 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=31f062e6d6eacf38c98ea51ae134e46f, UNASSIGN in 1.5190 sec 2023-04-10 02:56:13,448 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095373448"}]},"ts":"1681095373448"} 2023-04-10 02:56:13,449 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-10 02:56:13,450 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-10 02:56:13,451 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5300 sec 2023-04-10 02:56:14,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-10 02:56:14,028 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-10 02:56:14,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-10 02:56:14,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:14,031 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:14,032 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:14,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-10 02:56:14,036 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:14,038 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/recovered.edits] 2023-04-10 02:56:14,041 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/03b718a8f7ab4427a0433def85ed2b82 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/03b718a8f7ab4427a0433def85ed2b82 2023-04-10 02:56:14,043 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/3f509059fa774264a2ecfea4410f47a6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/3f509059fa774264a2ecfea4410f47a6 2023-04-10 02:56:14,044 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/c1ffb93df2174d468d0ebbeaf509234f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/A/c1ffb93df2174d468d0ebbeaf509234f 2023-04-10 02:56:14,046 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/506c1778f2644862864a1d391c1b15d3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/506c1778f2644862864a1d391c1b15d3 2023-04-10 02:56:14,047 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/afd5a2a653bd40cda2d6f2b8d6c16d37 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/afd5a2a653bd40cda2d6f2b8d6c16d37 2023-04-10 02:56:14,048 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/e78d96c3c70247c69c6976e92accb0ab to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/B/e78d96c3c70247c69c6976e92accb0ab 2023-04-10 02:56:14,050 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/213547a6db3c4dcfa87fe94afa761fda to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/213547a6db3c4dcfa87fe94afa761fda 2023-04-10 02:56:14,051 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/41d19752940c49a38685bcd343727be1 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/41d19752940c49a38685bcd343727be1 2023-04-10 02:56:14,052 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/4ac7405b084341f38ef1e920f693b38f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/4ac7405b084341f38ef1e920f693b38f 2023-04-10 02:56:14,053 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f3f8237032e4490fb6d2dc0eb3e79e1e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/C/f3f8237032e4490fb6d2dc0eb3e79e1e 2023-04-10 02:56:14,057 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/recovered.edits/629.seqid to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f/recovered.edits/629.seqid 2023-04-10 02:56:14,058 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/31f062e6d6eacf38c98ea51ae134e46f 2023-04-10 02:56:14,058 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-10 02:56:14,060 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:14,061 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-10 02:56:14,063 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-10 02:56:14,064 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:14,064 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-10 02:56:14,064 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681095374064"}]},"ts":"9223372036854775807"} 2023-04-10 02:56:14,065 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-10 02:56:14,065 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 31f062e6d6eacf38c98ea51ae134e46f, NAME => 'TestAcidGuarantees,,1681095346520.31f062e6d6eacf38c98ea51ae134e46f.', STARTKEY => '', ENDKEY => ''}] 2023-04-10 02:56:14,065 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-10 02:56:14,066 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681095374065"}]},"ts":"9223372036854775807"} 2023-04-10 02:56:14,067 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-10 02:56:14,068 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:14,069 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 40 msec 2023-04-10 02:56:14,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-10 02:56:14,134 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-10 02:56:14,146 INFO [Listener at localhost/44533] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227 (was 235), OpenFileDescriptor=561 (was 552) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=463 (was 417) - SystemLoadAverage LEAK? -, ProcessCount=180 (was 184), AvailableMemoryMB=2375 (was 1319) - AvailableMemoryMB LEAK? - 2023-04-10 02:56:14,159 INFO [Listener at localhost/44533] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227, OpenFileDescriptor=561, MaxFileDescriptor=60000, SystemLoadAverage=463, ProcessCount=180, AvailableMemoryMB=2375 2023-04-10 02:56:14,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-10 02:56:14,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-10 02:56:14,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:14,163 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-10 02:56:14,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-10 02:56:14,164 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-10 02:56:14,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-10 02:56:14,165 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:14,165 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7 empty. 2023-04-10 02:56:14,166 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:14,166 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-10 02:56:14,175 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-10 02:56:14,176 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6627f276a3252fdcc4f8b38ace724cf7, NAME => 'TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp 2023-04-10 02:56:14,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-10 02:56:14,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-10 02:56:14,581 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:56:14,582 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 6627f276a3252fdcc4f8b38ace724cf7, disabling compactions & flushes 2023-04-10 02:56:14,582 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:14,582 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:14,582 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. after waiting 0 ms 2023-04-10 02:56:14,582 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:14,582 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:14,582 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:14,584 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-10 02:56:14,585 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095374584"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095374584"}]},"ts":"1681095374584"} 2023-04-10 02:56:14,585 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-10 02:56:14,586 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-10 02:56:14,586 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095374586"}]},"ts":"1681095374586"} 2023-04-10 02:56:14,587 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-10 02:56:14,590 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6627f276a3252fdcc4f8b38ace724cf7, ASSIGN}] 2023-04-10 02:56:14,592 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6627f276a3252fdcc4f8b38ace724cf7, ASSIGN 2023-04-10 02:56:14,593 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6627f276a3252fdcc4f8b38ace724cf7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38321,1681095222501; forceNewPlan=false, retain=false 2023-04-10 02:56:14,744 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=6627f276a3252fdcc4f8b38ace724cf7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:14,744 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095374744"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095374744"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095374744"}]},"ts":"1681095374744"} 2023-04-10 02:56:14,746 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:56:14,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-10 02:56:14,900 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:14,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6627f276a3252fdcc4f8b38ace724cf7, NAME => 'TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.', STARTKEY => '', ENDKEY => ''} 2023-04-10 02:56:14,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:14,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:56:14,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:14,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:14,902 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:14,903 DEBUG [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A 2023-04-10 02:56:14,904 DEBUG [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A 2023-04-10 02:56:14,904 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:56:14,904 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6627f276a3252fdcc4f8b38ace724cf7 columnFamilyName A 2023-04-10 02:56:14,905 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.HStore(310): Store=6627f276a3252fdcc4f8b38ace724cf7/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:56:14,905 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:14,906 DEBUG [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B 2023-04-10 02:56:14,906 DEBUG [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B 2023-04-10 02:56:14,906 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:56:14,906 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6627f276a3252fdcc4f8b38ace724cf7 columnFamilyName B 2023-04-10 02:56:14,907 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.HStore(310): Store=6627f276a3252fdcc4f8b38ace724cf7/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:56:14,907 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:14,908 DEBUG [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C 2023-04-10 02:56:14,908 DEBUG [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C 2023-04-10 02:56:14,908 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:56:14,909 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6627f276a3252fdcc4f8b38ace724cf7 columnFamilyName C 2023-04-10 02:56:14,909 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.HStore(310): Store=6627f276a3252fdcc4f8b38ace724cf7/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:56:14,909 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:14,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:14,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:14,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-10 02:56:14,918 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:14,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-10 02:56:14,921 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6627f276a3252fdcc4f8b38ace724cf7; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61673971, jitterRate=-0.08098621666431427}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-10 02:56:14,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:14,921 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., pid=56, masterSystemTime=1681095374897 2023-04-10 02:56:14,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:14,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:14,923 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=6627f276a3252fdcc4f8b38ace724cf7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:14,923 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095374923"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681095374923"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681095374923"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681095374923"}]},"ts":"1681095374923"} 2023-04-10 02:56:14,926 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-10 02:56:14,926 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 in 179 msec 2023-04-10 02:56:14,927 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-10 02:56:14,928 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6627f276a3252fdcc4f8b38ace724cf7, ASSIGN in 336 msec 2023-04-10 02:56:14,928 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-10 02:56:14,928 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095374928"}]},"ts":"1681095374928"} 2023-04-10 02:56:14,929 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-10 02:56:14,931 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-10 02:56:14,932 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 770 msec 2023-04-10 02:56:15,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-10 02:56:15,268 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-10 02:56:15,269 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x6f06dd50 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:56:15,273 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59185ba7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:56:15,275 DEBUG [Listener at localhost/44533] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:56:15,276 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58968, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:56:15,277 DEBUG [Listener at localhost/44533] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-10 02:56:15,279 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-10 02:56:15,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-10 02:56:15,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-10 02:56:15,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:15,691 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-10 02:56:15,692 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-10 02:56:15,692 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-10 02:56:15,694 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-10 02:56:15,696 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6627f276a3252fdcc4f8b38ace724cf7, REOPEN/MOVE}] 2023-04-10 02:56:15,697 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6627f276a3252fdcc4f8b38ace724cf7, REOPEN/MOVE 2023-04-10 02:56:15,697 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=6627f276a3252fdcc4f8b38ace724cf7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:15,698 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095375697"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095375697"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095375697"}]},"ts":"1681095375697"} 2023-04-10 02:56:15,699 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:56:15,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:15,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6627f276a3252fdcc4f8b38ace724cf7, disabling compactions & flushes 2023-04-10 02:56:15,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:15,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:15,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. after waiting 0 ms 2023-04-10 02:56:15,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:15,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-10 02:56:15,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:15,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:15,855 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 6627f276a3252fdcc4f8b38ace724cf7 to self. 2023-04-10 02:56:15,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:15,857 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=6627f276a3252fdcc4f8b38ace724cf7, regionState=CLOSED 2023-04-10 02:56:15,857 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095375857"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095375857"}]},"ts":"1681095375857"} 2023-04-10 02:56:15,859 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-10 02:56:15,859 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 in 160 msec 2023-04-10 02:56:15,859 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6627f276a3252fdcc4f8b38ace724cf7, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,38321,1681095222501; forceNewPlan=false, retain=true 2023-04-10 02:56:16,010 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=6627f276a3252fdcc4f8b38ace724cf7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,010 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095376010"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095376010"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095376010"}]},"ts":"1681095376010"} 2023-04-10 02:56:16,011 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:56:16,166 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:16,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6627f276a3252fdcc4f8b38ace724cf7, NAME => 'TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.', STARTKEY => '', ENDKEY => ''} 2023-04-10 02:56:16,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:16,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-10 02:56:16,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:16,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:16,168 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:16,169 DEBUG [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A 2023-04-10 02:56:16,169 DEBUG [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A 2023-04-10 02:56:16,169 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:56:16,169 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6627f276a3252fdcc4f8b38ace724cf7 columnFamilyName A 2023-04-10 02:56:16,170 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.HStore(310): Store=6627f276a3252fdcc4f8b38ace724cf7/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:56:16,171 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:16,172 DEBUG [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B 2023-04-10 02:56:16,172 DEBUG [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B 2023-04-10 02:56:16,172 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:56:16,172 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6627f276a3252fdcc4f8b38ace724cf7 columnFamilyName B 2023-04-10 02:56:16,173 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.HStore(310): Store=6627f276a3252fdcc4f8b38ace724cf7/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:56:16,173 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:16,173 DEBUG [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C 2023-04-10 02:56:16,174 DEBUG [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C 2023-04-10 02:56:16,174 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-10 02:56:16,174 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6627f276a3252fdcc4f8b38ace724cf7 columnFamilyName C 2023-04-10 02:56:16,174 INFO [StoreOpener-6627f276a3252fdcc4f8b38ace724cf7-1] regionserver.HStore(310): Store=6627f276a3252fdcc4f8b38ace724cf7/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-10 02:56:16,175 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:16,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:16,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:16,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-10 02:56:16,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:16,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6627f276a3252fdcc4f8b38ace724cf7; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73844965, jitterRate=0.10037572681903839}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-10 02:56:16,179 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:16,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., pid=61, masterSystemTime=1681095376163 2023-04-10 02:56:16,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:16,181 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:16,181 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=6627f276a3252fdcc4f8b38ace724cf7, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,181 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095376181"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681095376181"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681095376181"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681095376181"}]},"ts":"1681095376181"} 2023-04-10 02:56:16,183 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-10 02:56:16,183 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 in 171 msec 2023-04-10 02:56:16,185 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-10 02:56:16,185 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6627f276a3252fdcc4f8b38ace724cf7, REOPEN/MOVE in 487 msec 2023-04-10 02:56:16,187 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-10 02:56:16,187 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 491 msec 2023-04-10 02:56:16,188 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 907 msec 2023-04-10 02:56:16,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-10 02:56:16,190 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e8654 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:56:16,196 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62b3caa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:56:16,196 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x4db22237 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:56:16,204 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b5b1f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:56:16,205 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x08b57aae to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:56:16,212 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@544a4215, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:56:16,212 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x06799455 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:56:16,219 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25baee3e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:56:16,219 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x66c25427 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:56:16,227 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c53bf96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:56:16,227 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x3b59bdcf to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:56:16,243 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a470a3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:56:16,243 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x3d765193 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:56:16,251 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bb72437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:56:16,251 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x32bf16cd to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:56:16,261 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dca23c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:56:16,261 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x28164190 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:56:16,269 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62d52321, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:56:16,269 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(139): Connect 0x308b3f39 to 127.0.0.1:60568 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-10 02:56:16,304 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a8bf279, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-10 02:56:16,320 DEBUG [hconnection-0x3850d5ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:56:16,320 DEBUG [hconnection-0x6c5a9742-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:56:16,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:16,328 DEBUG [hconnection-0x6d9fba23-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:56:16,328 DEBUG [hconnection-0x65a7bbe6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:56:16,328 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58984, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:56:16,328 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58992, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:56:16,328 DEBUG [hconnection-0x2abeeea9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:56:16,328 DEBUG [hconnection-0x1346db06-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:56:16,328 DEBUG [hconnection-0x662cd978-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:56:16,328 DEBUG [hconnection-0x3e47483b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:56:16,328 DEBUG [hconnection-0x4e0ba007-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:56:16,330 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59000, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:56:16,336 DEBUG [hconnection-0x4795d546-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-10 02:56:16,336 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59046, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:56:16,336 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59030, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:56:16,337 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59006, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:56:16,339 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59054, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:56:16,339 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59034, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:56:16,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:16,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:16,344 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:16,344 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:16,344 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:16,344 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:16,349 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59014, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:56:16,353 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:16,353 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:16,352 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59056, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-10 02:56:16,356 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:16,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:16,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:16,357 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:16,357 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:16,357 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:16,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:16,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:16,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:16,361 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:16,361 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:16,361 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:16,361 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:16,362 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-10 02:56:16,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:16,362 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:16,363 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:16,363 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:16,363 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:16,363 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:16,363 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:16,363 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:16,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095376376, max timestamp=1681095376379Num uniques 859; 2023-04-10 02:56:16,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:16,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095376376, max timestamp=1681095376379Num uniques 859; 2023-04-10 02:56:16,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:16,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095376376, max timestamp=1681095376379Num uniques 859; 2023-04-10 02:56:16,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:16,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095436398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095436400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095436400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681095376389, max timestamp=1681095376402Num uniques 1018; 2023-04-10 02:56:16,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-10 02:56:16,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681095376396, max timestamp=1681095376402Num uniques 1018; 2023-04-10 02:56:16,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-10 02:56:16,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681095376396, max timestamp=1681095376402Num uniques 1018; 2023-04-10 02:56:16,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-10 02:56:16,414 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041051a1bed015d742daabcfbcab44b5cc04 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041051a1bed015d742daabcfbcab44b5cc04 2023-04-10 02:56:16,442 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/ba9d3ac58c2f4b7bb923548686281c4d 2023-04-10 02:56:16,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58984 deadline: 1681095436439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59046 deadline: 1681095436439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,483 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/3cc63c4c175f413bbd06d9243099b29a 2023-04-10 02:56:16,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095436507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095436509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095436510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58984 deadline: 1681095436543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59046 deadline: 1681095436543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095436709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095436711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095436712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59046 deadline: 1681095436745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:16,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58984 deadline: 1681095436746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,902 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/b9c48bd3b53c4fcba79b854a519a0942 2023-04-10 02:56:16,906 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/ba9d3ac58c2f4b7bb923548686281c4d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ba9d3ac58c2f4b7bb923548686281c4d 2023-04-10 02:56:16,911 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ba9d3ac58c2f4b7bb923548686281c4d, entries=150, sequenceid=14, filesize=25.1 K 2023-04-10 02:56:16,912 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/3cc63c4c175f413bbd06d9243099b29a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/3cc63c4c175f413bbd06d9243099b29a 2023-04-10 02:56:16,915 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/3cc63c4c175f413bbd06d9243099b29a, entries=150, sequenceid=14, filesize=11.6 K 2023-04-10 02:56:16,916 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/b9c48bd3b53c4fcba79b854a519a0942 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/b9c48bd3b53c4fcba79b854a519a0942 2023-04-10 02:56:16,920 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/b9c48bd3b53c4fcba79b854a519a0942, entries=150, sequenceid=14, filesize=11.6 K 2023-04-10 02:56:16,923 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for 6627f276a3252fdcc4f8b38ace724cf7 in 561ms, sequenceid=14, compaction requested=false 2023-04-10 02:56:16,923 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-10 02:56:16,925 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:16,925 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:16,928 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:16,928 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:16,928 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,928 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:16,928 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:16,937 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:16,938 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,938 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:16,939 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:16,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:16,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:16,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d7c697[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:16,939 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:16,939 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:16,942 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:16,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:16,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:16,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:16,942 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:16,942 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,942 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:16,943 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,943 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:16,944 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:16,944 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:16,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:16,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:16,947 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:16,947 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:16,947 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:16,947 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:16,947 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:16,951 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:16,952 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:16,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:16,952 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:16,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:16,952 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:16,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,955 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:16,955 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:16,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:16,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:16,955 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:16,955 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:16,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:16,956 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:16,956 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:16,955 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:16,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:16,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:16,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:16,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:16,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:16,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:16,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:16,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:16,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:17,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:17,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=143.13 KB heapSize=409.84 KB 2023-04-10 02:56:17,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:17,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:56:17,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:17,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:56:17,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:17,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:56:17,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095437025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095437026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095437026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,053 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410d605696ec40b4b40bc3f233f663851e9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410d605696ec40b4b40bc3f233f663851e9 2023-04-10 02:56:17,056 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:17,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59046 deadline: 1681095437055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58984 deadline: 1681095437060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:17,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:17,093 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/b717eb6b935c448b96519483def42a9c 2023-04-10 02:56:17,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:17,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:17,100 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:17,100 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,100 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:17,100 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:17,101 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,101 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,103 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,103 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,103 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,103 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:17,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:17,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:17,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:17,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:17,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:17,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:17,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:17,109 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,109 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:17,109 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:17,109 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:17,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:17,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:17,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:17,111 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,111 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,111 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,111 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:17,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ae979c7[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:17,114 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:17,114 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,115 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,115 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:17,115 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:17,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:17,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:17,117 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:17,117 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,117 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:17,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:17,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:17,121 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:17,122 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:17,122 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:17,122 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:17,122 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:17,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,125 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,128 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,131 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,131 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,131 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,131 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,131 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,131 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,131 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,131 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,131 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,131 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:17,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:17,132 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:17,132 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:17,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:17,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095437130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095437130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095437130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/b12e04768bce4f6e825ad839293be42d 2023-04-10 02:56:17,232 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:17,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:17,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:17,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:17,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:17,240 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:17,240 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,240 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:17,240 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:17,241 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,241 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,242 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,243 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:17,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:17,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:17,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:17,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:17,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,252 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:17,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:17,260 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:17,264 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,264 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:17,264 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:17,264 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,264 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:17,264 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:17,264 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,264 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:17,264 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:17,269 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,269 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,270 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,270 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:17,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52e7707c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:17,272 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:17,273 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,274 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,274 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:17,274 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:17,274 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:17,274 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,274 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:17,276 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:17,276 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,276 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,276 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:17,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:17,280 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:17,280 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:17,280 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:17,280 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:17,280 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:17,285 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,285 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,286 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,287 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,289 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,292 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,292 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,293 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,293 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,293 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,293 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,293 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:17,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:17,293 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,293 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,293 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:17,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,293 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:17,294 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:17,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095437333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095437335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095437335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,394 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:17,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:17,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:17,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:17,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:17,397 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:17,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:17,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:17,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,398 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,398 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,399 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:17,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:17,402 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:17,402 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,402 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:17,403 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:17,403 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:17,403 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,403 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:17,403 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:17,403 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,403 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:17,403 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:17,403 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,403 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:17,403 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:17,405 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,406 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,406 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:17,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35c98c32[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:17,410 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:17,410 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,411 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,412 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,412 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:17,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:17,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:17,412 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:17,414 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:17,414 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,414 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,414 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:17,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:17,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:17,418 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:17,418 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:17,418 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:17,418 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:17,418 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:17,420 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,420 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,420 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,421 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,423 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,427 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:17,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:17,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,428 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:17,428 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:17,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,528 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:17,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:17,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:17,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:17,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:17,540 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:17,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:17,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:17,541 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,541 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,544 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,544 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,545 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,545 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:17,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:17,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:17,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:17,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:17,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:17,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:17,560 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,560 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:17,560 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:17,560 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:17,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:17,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:17,560 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:17,562 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/9bf36dcb425846df93fc0df4ad232626 2023-04-10 02:56:17,563 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,563 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,563 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,564 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:17,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59046 deadline: 1681095437562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:17,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39260762[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:17,566 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:17,566 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/b717eb6b935c448b96519483def42a9c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/b717eb6b935c448b96519483def42a9c 2023-04-10 02:56:17,567 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,567 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:17,567 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:17,567 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:17,567 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,567 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:17,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58984 deadline: 1681095437568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,569 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:17,570 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,570 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:17,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/b717eb6b935c448b96519483def42a9c, entries=150, sequenceid=69, filesize=25.1 K 2023-04-10 02:56:17,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/b12e04768bce4f6e825ad839293be42d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b12e04768bce4f6e825ad839293be42d 2023-04-10 02:56:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:17,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:17,575 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:17,575 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:17,575 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:17,575 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:17,575 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:17,577 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,577 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,577 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,579 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b12e04768bce4f6e825ad839293be42d, entries=150, sequenceid=69, filesize=11.6 K 2023-04-10 02:56:17,580 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/9bf36dcb425846df93fc0df4ad232626 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/9bf36dcb425846df93fc0df4ad232626 2023-04-10 02:56:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,581 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,585 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,585 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,585 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,585 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,585 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,585 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:17,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:17,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:17,585 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,585 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:17,585 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:17,585 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,585 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,585 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/9bf36dcb425846df93fc0df4ad232626, entries=150, sequenceid=69, filesize=11.6 K 2023-04-10 02:56:17,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,586 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~409.80 KB/419632, currentSize=40.25 KB/41220 for 6627f276a3252fdcc4f8b38ace724cf7 in 570ms, sequenceid=69, compaction requested=false 2023-04-10 02:56:17,587 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:17,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:17,640 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:56:17,641 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:17,641 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:17,641 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:17,641 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:17,641 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:17,641 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:17,663 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410bf71920d739f4c3d84d21299adb7e918 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410bf71920d739f4c3d84d21299adb7e918 2023-04-10 02:56:17,685 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:17,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:17,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:17,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:17,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:17,691 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:17,691 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,691 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:17,691 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:17,691 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,691 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,693 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,693 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,693 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:17,693 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:17,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:17,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:17,697 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,697 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:17,697 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:17,697 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:17,699 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,699 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,700 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,700 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:17,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095377691, max timestamp=1681095377698Num uniques 859; 2023-04-10 02:56:17,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:17,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:17,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18a076c0[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:17,701 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:17,701 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095377691, max timestamp=1681095377699Num uniques 859; 2023-04-10 02:56:17,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:17,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095377691, max timestamp=1681095377698Num uniques 859; 2023-04-10 02:56:17,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:17,706 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,706 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,706 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:17,708 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:17,708 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:17,708 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:17,709 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:17,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:17,713 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:17,713 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:17,713 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:17,713 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:17,714 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:17,715 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,715 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,715 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,716 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,718 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,721 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,721 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,721 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:17,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:17,722 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:17,722 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:17,721 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,722 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,721 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,721 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,722 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,722 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,722 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681095377712, max timestamp=1681095377730Num uniques 1018; 2023-04-10 02:56:17,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-10 02:56:17,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681095377712, max timestamp=1681095377730Num uniques 1018; 2023-04-10 02:56:17,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-10 02:56:17,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681095377712, max timestamp=1681095377730Num uniques 1018; 2023-04-10 02:56:17,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-10 02:56:17,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095437761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095437761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095437764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,822 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:17,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:17,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:17,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:17,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:17,825 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:17,826 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,826 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:17,826 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:17,826 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,826 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,830 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,831 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,831 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:17,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:17,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:17,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:17,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,842 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:17,842 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:17,842 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:17,842 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,842 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:17,842 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:17,842 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,842 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:17,842 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:17,845 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,845 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,848 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,848 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:17,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@712d8138[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:17,850 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:17,850 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,851 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,851 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,852 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:17,852 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:17,852 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:17,852 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:17,853 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:17,853 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,853 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:17,855 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:17,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:17,858 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:17,858 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:17,858 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:17,858 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:17,858 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:17,860 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,860 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,860 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,862 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,865 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,865 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,865 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,865 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,866 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,866 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,866 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:17,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:17,865 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,866 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:17,866 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:17,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,866 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,866 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:17,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095437866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095437866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:17,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095437869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,966 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:17,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:17,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:17,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:17,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:17,969 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:17,969 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,970 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:17,970 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:17,970 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,970 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,971 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,971 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:17,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:17,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:17,976 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:17,976 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:17,976 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:17,977 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:17,977 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:17,978 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,978 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:17,978 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:17,978 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:17,978 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:17,978 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:17,978 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,978 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:17,978 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:17,978 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,978 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:17,979 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,979 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,980 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,980 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:17,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:17,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@87c2b33[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:17,981 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:17,981 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,983 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,983 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:17,983 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:17,983 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:17,983 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,983 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:17,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:17,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,985 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,985 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:17,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:17,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:17,989 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:17,989 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:17,989 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:17,989 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:17,989 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:17,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:17,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,991 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:17,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,993 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:17,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:17,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,995 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:17,995 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:17,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:17,996 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,996 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,996 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:17,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:17,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:17,996 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:17,996 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:17,996 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,996 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:17,996 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:17,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:17,996 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:17,996 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:18,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095438071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095438071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,075 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=79, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/dbc258adb7cb423eb114c6c3ec3005af 2023-04-10 02:56:18,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095438075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/7d171740fec54061b76b18bb8f4407d8 2023-04-10 02:56:18,096 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:18,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:18,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:18,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:18,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:18,104 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:18,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:18,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:18,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:18,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/143aa7833e6e4a34aeffa274ff702a46 2023-04-10 02:56:18,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:18,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:18,106 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:18,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:18,106 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:18,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:18,107 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,107 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:18,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/dbc258adb7cb423eb114c6c3ec3005af as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/dbc258adb7cb423eb114c6c3ec3005af 2023-04-10 02:56:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:18,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:18,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:18,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:18,108 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:18,108 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:18,108 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:18,109 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:18,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:18,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:18,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:18,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:18,111 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/dbc258adb7cb423eb114c6c3ec3005af, entries=200, sequenceid=79, filesize=31.9 K 2023-04-10 02:56:18,111 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,111 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:18,111 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/7d171740fec54061b76b18bb8f4407d8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/7d171740fec54061b76b18bb8f4407d8 2023-04-10 02:56:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:18,112 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:18,112 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:18,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:18,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e82049[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:18,113 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:18,113 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:18,115 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/7d171740fec54061b76b18bb8f4407d8, entries=150, sequenceid=79, filesize=11.6 K 2023-04-10 02:56:18,115 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:18,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:18,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:18,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:18,115 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:18,115 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,115 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/143aa7833e6e4a34aeffa274ff702a46 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/143aa7833e6e4a34aeffa274ff702a46 2023-04-10 02:56:18,115 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:18,117 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,117 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:18,117 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:18,117 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:18,119 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/143aa7833e6e4a34aeffa274ff702a46, entries=150, sequenceid=79, filesize=11.6 K 2023-04-10 02:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:18,120 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6627f276a3252fdcc4f8b38ace724cf7 in 480ms, sequenceid=79, compaction requested=true 2023-04-10 02:56:18,120 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:18,120 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:18,120 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:18,120 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:18,120 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:18,120 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:18,120 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:18,120 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:18,120 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:18,120 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:18,120 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:18,121 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:18,122 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/A is initiating minor compaction (all files) 2023-04-10 02:56:18,122 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/A in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:18,122 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ba9d3ac58c2f4b7bb923548686281c4d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/b717eb6b935c448b96519483def42a9c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/dbc258adb7cb423eb114c6c3ec3005af] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=82.2 K 2023-04-10 02:56:18,122 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:18,122 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/B is initiating minor compaction (all files) 2023-04-10 02:56:18,122 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/B in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:18,122 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/3cc63c4c175f413bbd06d9243099b29a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b12e04768bce4f6e825ad839293be42d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/7d171740fec54061b76b18bb8f4407d8] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=34.8 K 2023-04-10 02:56:18,123 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ba9d3ac58c2f4b7bb923548686281c4d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681095376355 2023-04-10 02:56:18,123 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 3cc63c4c175f413bbd06d9243099b29a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681095376355 2023-04-10 02:56:18,123 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting b717eb6b935c448b96519483def42a9c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681095376435 2023-04-10 02:56:18,123 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting b12e04768bce4f6e825ad839293be42d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681095376435 2023-04-10 02:56:18,123 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting dbc258adb7cb423eb114c6c3ec3005af, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681095377019 2023-04-10 02:56:18,124 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 7d171740fec54061b76b18bb8f4407d8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681095377019 2023-04-10 02:56:18,127 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:18,127 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:18,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:18,127 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:18,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:18,130 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:18,132 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,136 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:18,136 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,136 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:18,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:18,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:18,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:18,136 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:18,136 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:18,136 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:18,136 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,136 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:18,136 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:18,136 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:18,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:18,136 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:18,137 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:18,141 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#B#compaction#508 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:18,149 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#A#compaction#509 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:18,237 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:18,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:18,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:18,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:18,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:18,240 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:18,240 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:18,240 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:18,240 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:18,240 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:18,241 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:18,242 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:18,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:18,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,242 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:18,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:18,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:18,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:18,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:18,243 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:18,243 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:18,243 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:18,244 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:18,244 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:18,244 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=134.18 KB heapSize=384.14 KB 2023-04-10 02:56:18,244 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:18,244 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:18,244 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:18,244 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:18,244 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:18,245 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:18,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:18,376 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:18,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095438406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095438412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095438412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095438513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095438517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095438517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,564 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/ed46b134a7714c69ba9481cc3dae5f7f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/ed46b134a7714c69ba9481cc3dae5f7f 2023-04-10 02:56:18,575 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/B of 6627f276a3252fdcc4f8b38ace724cf7 into ed46b134a7714c69ba9481cc3dae5f7f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:18,575 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:18,575 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/B, priority=13, startTime=1681095378120; duration=0sec 2023-04-10 02:56:18,575 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:18,575 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:18,576 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:18,576 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/C is initiating minor compaction (all files) 2023-04-10 02:56:18,576 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/C in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:18,577 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/b9c48bd3b53c4fcba79b854a519a0942, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/9bf36dcb425846df93fc0df4ad232626, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/143aa7833e6e4a34aeffa274ff702a46] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=34.8 K 2023-04-10 02:56:18,577 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting b9c48bd3b53c4fcba79b854a519a0942, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681095376355 2023-04-10 02:56:18,577 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 9bf36dcb425846df93fc0df4ad232626, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681095376435 2023-04-10 02:56:18,578 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 143aa7833e6e4a34aeffa274ff702a46, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681095377019 2023-04-10 02:56:18,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59046 deadline: 1681095438577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58984 deadline: 1681095438578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,595 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#C#compaction#511 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:18,656 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304106448505a9f234f31bd2c4e52de283ed8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304106448505a9f234f31bd2c4e52de283ed8 2023-04-10 02:56:18,663 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=133, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/935e23ce0b3547a387d1ae8272b9d88f 2023-04-10 02:56:18,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095438716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095438719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:18,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095438721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:18,973 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-10 02:56:18,983 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/a869565cbedb436b8093ea399e6a8774 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a869565cbedb436b8093ea399e6a8774 2023-04-10 02:56:18,987 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/A of 6627f276a3252fdcc4f8b38ace724cf7 into a869565cbedb436b8093ea399e6a8774(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:18,987 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:18,987 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/A, priority=13, startTime=1681095378120; duration=0sec 2023-04-10 02:56:18,987 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:19,008 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/0029e1d267ce420f8afe97b7a261cd5e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0029e1d267ce420f8afe97b7a261cd5e 2023-04-10 02:56:19,011 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/C of 6627f276a3252fdcc4f8b38ace724cf7 into 0029e1d267ce420f8afe97b7a261cd5e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:19,011 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:19,011 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/C, priority=13, startTime=1681095378120; duration=0sec 2023-04-10 02:56:19,011 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:19,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:19,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095439020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:19,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095439022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:19,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095439023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,074 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/bd434acadcb14115acbe0c01dee9f6bc 2023-04-10 02:56:19,084 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/0a65f2b5736d4cedaa62c102b2397507 2023-04-10 02:56:19,088 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/935e23ce0b3547a387d1ae8272b9d88f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/935e23ce0b3547a387d1ae8272b9d88f 2023-04-10 02:56:19,092 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/935e23ce0b3547a387d1ae8272b9d88f, entries=150, sequenceid=133, filesize=25.2 K 2023-04-10 02:56:19,093 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/bd434acadcb14115acbe0c01dee9f6bc as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/bd434acadcb14115acbe0c01dee9f6bc 2023-04-10 02:56:19,096 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/bd434acadcb14115acbe0c01dee9f6bc, entries=150, sequenceid=133, filesize=11.7 K 2023-04-10 02:56:19,097 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/0a65f2b5736d4cedaa62c102b2397507 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0a65f2b5736d4cedaa62c102b2397507 2023-04-10 02:56:19,100 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0a65f2b5736d4cedaa62c102b2397507, entries=150, sequenceid=133, filesize=11.7 K 2023-04-10 02:56:19,101 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.09 KB/393312, currentSize=53.67 KB/54960 for 6627f276a3252fdcc4f8b38ace724cf7 in 857ms, sequenceid=133, compaction requested=false 2023-04-10 02:56:19,101 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:19,101 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:19,101 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:19,101 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:19,101 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,101 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:19,101 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:19,103 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,103 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:19,103 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,103 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:19,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:19,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ae550c2[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:19,105 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:19,105 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,107 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:19,108 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,108 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:19,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:19,109 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:19,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:19,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:19,112 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:19,112 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:19,113 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:19,113 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:19,113 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:19,114 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,114 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,114 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:19,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:19,115 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:19,116 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,118 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,119 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:19,119 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:19,119 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,119 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:19,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:19,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:19,119 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:19,119 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:19,119 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:19,119 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:19,119 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:19,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:19,119 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,119 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:19,219 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:19,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:19,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:19,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:19,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:19,230 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:19,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:19,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:19,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:19,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,231 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:19,233 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:19,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:19,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:19,233 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:19,233 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,233 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:19,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:19,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:19,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:19,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:19,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:19,248 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:19,248 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:19,249 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:56:19,249 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:19,249 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:19,249 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:19,249 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:19,249 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:19,249 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:19,281 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410620483d45a5d428b9ce14f4c70779fb7 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410620483d45a5d428b9ce14f4c70779fb7 2023-04-10 02:56:19,294 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/f0da119f606b4fd29843a4568c0daefe 2023-04-10 02:56:19,312 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/c0bf5c2122e04a589cd1e3eef2b0ba28 2023-04-10 02:56:19,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:19,526 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:19,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095379537, max timestamp=1681095379550Num uniques 859; 2023-04-10 02:56:19,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095379537, max timestamp=1681095379550Num uniques 859; 2023-04-10 02:56:19,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:19,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681095379537, max timestamp=1681095379550Num uniques 859; 2023-04-10 02:56:19,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-10 02:56:19,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:19,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:19,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095439569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:19,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095439570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:19,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095439571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:19,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:19,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095439673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095439673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:19,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095439674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,722 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/a3c140390ce14fdca6ef3ce5e3098d71 2023-04-10 02:56:19,725 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/f0da119f606b4fd29843a4568c0daefe as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f0da119f606b4fd29843a4568c0daefe 2023-04-10 02:56:19,729 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f0da119f606b4fd29843a4568c0daefe, entries=150, sequenceid=147, filesize=25.3 K 2023-04-10 02:56:19,729 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/c0bf5c2122e04a589cd1e3eef2b0ba28 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/c0bf5c2122e04a589cd1e3eef2b0ba28 2023-04-10 02:56:19,732 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/c0bf5c2122e04a589cd1e3eef2b0ba28, entries=150, sequenceid=147, filesize=11.8 K 2023-04-10 02:56:19,733 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/a3c140390ce14fdca6ef3ce5e3098d71 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a3c140390ce14fdca6ef3ce5e3098d71 2023-04-10 02:56:19,736 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a3c140390ce14fdca6ef3ce5e3098d71, entries=150, sequenceid=147, filesize=11.8 K 2023-04-10 02:56:19,736 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.08 KB/130125 for 6627f276a3252fdcc4f8b38ace724cf7 in 487ms, sequenceid=147, compaction requested=true 2023-04-10 02:56:19,737 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:19,737 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:19,740 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:19,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,740 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:19,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:19,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75c26e7[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:19,742 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:19,742 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,744 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,744 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:19,744 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:19,744 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:19,744 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,744 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:19,745 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,745 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:19,745 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:19,745 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:19,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:19,748 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:19,748 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:19,748 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:19,748 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:19,748 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:19,750 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,750 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:19,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,751 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:19,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,752 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:19,753 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:19,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:19,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:19,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:19,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:19,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:19,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:19,755 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:19,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:19,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:19,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:19,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,756 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:19,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:19,755 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:19,756 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:19,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:19,756 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:19,756 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:19,856 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:19,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:19,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:19,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:19,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:19,860 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:19,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:19,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:19,860 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:19,861 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,861 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:19,862 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:19,862 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:19,863 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,863 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:19,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:19,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:19,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:19,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:19,864 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:19,864 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:19,864 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:19,864 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:19,865 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:19,865 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:19,865 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:19,865 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=127.08 KB heapSize=364.15 KB 2023-04-10 02:56:19,865 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:19,865 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:19,865 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:19,865 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:19,865 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:19,865 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:19,880 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041054831c19ae4f4a7589e392df015f18a0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041054831c19ae4f4a7589e392df015f18a0 2023-04-10 02:56:19,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:19,880 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:19,885 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=183, memsize=42.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/c285793ca63146238102dbc7da156144 2023-04-10 02:56:19,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:19,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095439893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:19,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095439893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:19,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:19,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095439899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095440000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095440000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095440002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095440205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095440205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095440206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,310 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/a4974822bade422d8b24be33eb50b375 2023-04-10 02:56:20,320 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/21f5816f0f684da4a6c4d0ae4fa9bc1a 2023-04-10 02:56:20,324 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/c285793ca63146238102dbc7da156144 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/c285793ca63146238102dbc7da156144 2023-04-10 02:56:20,327 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/c285793ca63146238102dbc7da156144, entries=150, sequenceid=183, filesize=25.3 K 2023-04-10 02:56:20,328 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/a4974822bade422d8b24be33eb50b375 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a4974822bade422d8b24be33eb50b375 2023-04-10 02:56:20,332 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a4974822bade422d8b24be33eb50b375, entries=150, sequenceid=183, filesize=11.8 K 2023-04-10 02:56:20,333 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/21f5816f0f684da4a6c4d0ae4fa9bc1a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/21f5816f0f684da4a6c4d0ae4fa9bc1a 2023-04-10 02:56:20,336 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/21f5816f0f684da4a6c4d0ae4fa9bc1a, entries=150, sequenceid=183, filesize=11.8 K 2023-04-10 02:56:20,336 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.08 KB/130125, heapSize ~364.10 KB/372840, currentSize=53.67 KB/54960 for 6627f276a3252fdcc4f8b38ace724cf7 in 471ms, sequenceid=183, compaction requested=true 2023-04-10 02:56:20,336 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:20,336 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:20,336 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:20,338 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,339 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,339 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:20,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:20,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e90b08c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:20,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:20,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,357 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,357 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:20,357 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:20,357 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:20,358 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,358 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:20,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:20,359 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:20,360 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:20,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:20,363 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:20,363 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:20,364 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:20,364 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:20,364 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:20,365 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,365 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:20,365 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:20,366 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:20,368 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:20,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:20,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:20,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:20,380 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:20,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:20,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:20,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:20,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:20,380 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:20,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:20,380 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:20,481 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:20,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:20,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:20,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:20,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:20,484 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:20,484 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:20,484 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:20,484 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:20,484 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,484 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:20,486 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:20,486 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:20,486 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,486 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:20,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:20,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:20,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:20,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:20,487 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:20,487 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:20,487 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:20,488 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:20,488 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:20,488 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:20,488 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:20,488 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:56:20,488 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:20,488 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:20,488 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:20,488 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:20,488 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:20,488 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:20,501 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410c83169eeac0246219b1c10f2f921dc78 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410c83169eeac0246219b1c10f2f921dc78 2023-04-10 02:56:20,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:20,509 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:20,510 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=194, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/3952d3e8be894c9186ea7ebfbbd2aefc 2023-04-10 02:56:20,520 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/a77a9a6a849a40a6bf4184a55b64db68 2023-04-10 02:56:20,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095440536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095380530, max timestamp=1681095380538Num uniques 859; 2023-04-10 02:56:20,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095380530, max timestamp=1681095380538Num uniques 859; 2023-04-10 02:56:20,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:56:20,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:56:20,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681095380530, max timestamp=1681095380538Num uniques 859; 2023-04-10 02:56:20,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-10 02:56:20,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095440570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095440575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59046 deadline: 1681095440587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,594 DEBUG [Thread-3757] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:56:20,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58984 deadline: 1681095440589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,596 DEBUG [Thread-3755] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:56:20,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095440641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095440676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095440680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095440848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095440880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:20,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095440885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,929 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/492b161d71ca497d8be324fce2c0be9b 2023-04-10 02:56:20,933 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/3952d3e8be894c9186ea7ebfbbd2aefc as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3952d3e8be894c9186ea7ebfbbd2aefc 2023-04-10 02:56:20,936 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3952d3e8be894c9186ea7ebfbbd2aefc, entries=150, sequenceid=194, filesize=25.3 K 2023-04-10 02:56:20,937 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/a77a9a6a849a40a6bf4184a55b64db68 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a77a9a6a849a40a6bf4184a55b64db68 2023-04-10 02:56:20,941 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a77a9a6a849a40a6bf4184a55b64db68, entries=150, sequenceid=194, filesize=11.8 K 2023-04-10 02:56:20,942 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/492b161d71ca497d8be324fce2c0be9b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/492b161d71ca497d8be324fce2c0be9b 2023-04-10 02:56:20,945 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/492b161d71ca497d8be324fce2c0be9b, entries=150, sequenceid=194, filesize=11.8 K 2023-04-10 02:56:20,946 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6627f276a3252fdcc4f8b38ace724cf7 in 458ms, sequenceid=194, compaction requested=true 2023-04-10 02:56:20,946 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:20,946 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:20,946 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:20,946 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:20,946 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,946 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:20,946 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:20,948 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,949 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:20,949 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:20,949 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:20,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:20,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eba2f7f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:20,951 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:20,951 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,952 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,952 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:20,952 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:20,952 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:20,952 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,952 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:20,954 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:20,955 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:20,955 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:20,955 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:20,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:20,957 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:20,957 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:20,957 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:20,957 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:20,957 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:20,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:20,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,959 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:20,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:20,961 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:20,961 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:20,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:20,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:20,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:20,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:20,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:20,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:20,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:20,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,964 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:20,964 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:20,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:20,965 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:20,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:20,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:21,065 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:21,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:21,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:21,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:21,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:21,070 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:21,070 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:21,070 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:21,070 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:21,071 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,071 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,072 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,072 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:21,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:21,072 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,072 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:21,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:21,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:21,074 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:21,074 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:21,074 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:21,075 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:21,075 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:21,075 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:21,075 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:21,075 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-10 02:56:21,075 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:21,075 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:21,075 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:21,076 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:21,076 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:21,076 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:21,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:21,156 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:21,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:21,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095441185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:21,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095441191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:21,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58992 deadline: 1681095441198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:21,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095441291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:21,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095441295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:21,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095441497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,502 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041045c4998d0c594d02af5bb484e199962b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041045c4998d0c594d02af5bb484e199962b 2023-04-10 02:56:21,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:21,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095441501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,509 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=229, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/a05d9c164b004ed7ad484e880568e9f1 2023-04-10 02:56:21,522 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/21edf2a3ef4849cc8122990aece622eb 2023-04-10 02:56:21,543 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/eb3222a0a97d4f31acb213494b13475e 2023-04-10 02:56:21,548 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/a05d9c164b004ed7ad484e880568e9f1 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a05d9c164b004ed7ad484e880568e9f1 2023-04-10 02:56:21,552 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a05d9c164b004ed7ad484e880568e9f1, entries=150, sequenceid=229, filesize=25.3 K 2023-04-10 02:56:21,553 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/21edf2a3ef4849cc8122990aece622eb as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/21edf2a3ef4849cc8122990aece622eb 2023-04-10 02:56:21,565 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/21edf2a3ef4849cc8122990aece622eb, entries=150, sequenceid=229, filesize=11.8 K 2023-04-10 02:56:21,566 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/eb3222a0a97d4f31acb213494b13475e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/eb3222a0a97d4f31acb213494b13475e 2023-04-10 02:56:21,571 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/eb3222a0a97d4f31acb213494b13475e, entries=150, sequenceid=229, filesize=11.8 K 2023-04-10 02:56:21,572 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 6627f276a3252fdcc4f8b38ace724cf7 in 497ms, sequenceid=229, compaction requested=true 2023-04-10 02:56:21,572 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:21,572 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:21,572 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:21,572 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:21,572 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,572 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:21,572 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:21,575 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:21,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,576 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:21,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:21,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e69092a[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:21,577 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:21,578 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,579 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:21,579 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,579 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:21,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:21,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:21,581 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:21,581 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:21,581 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,581 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:21,584 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:21,584 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:21,584 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:21,584 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:21,584 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:21,586 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,586 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:21,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,586 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:21,587 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,588 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,591 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:21,591 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:21,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:21,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:21,591 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,591 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:21,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:21,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:21,591 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:21,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,592 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,592 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:21,592 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:21,692 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:21,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:21,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:21,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:21,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:21,704 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:21,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:21,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:21,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:21,705 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,705 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,706 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,706 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:21,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:21,706 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,706 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:21,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:21,708 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:21,708 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:21,708 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:21,708 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:21,708 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:21,708 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:21,709 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:21,709 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:56:21,709 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:21,709 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:21,709 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:21,709 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:21,709 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:21,709 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:21,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:21,710 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:21,736 WARN [DataStreamer for file /user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410bc5ccf030d794d999460ae83ba60729e] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-10 02:56:21,741 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410bc5ccf030d794d999460ae83ba60729e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410bc5ccf030d794d999460ae83ba60729e 2023-04-10 02:56:21,766 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/e80a79a7ffa6424c88ca3856d6ad8eee 2023-04-10 02:56:21,777 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/52fd4ad790c04071992ad4289320486e 2023-04-10 02:56:21,797 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/fc48c85e63a7466da9bd4a66ac6999a9 2023-04-10 02:56:21,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:21,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095441801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095381767, max timestamp=1681095381803Num uniques 859; 2023-04-10 02:56:21,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:21,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095381767, max timestamp=1681095381803Num uniques 859; 2023-04-10 02:56:21,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:21,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095381767, max timestamp=1681095381803Num uniques 859; 2023-04-10 02:56:21,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:21,809 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/e80a79a7ffa6424c88ca3856d6ad8eee as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/e80a79a7ffa6424c88ca3856d6ad8eee 2023-04-10 02:56:21,813 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/e80a79a7ffa6424c88ca3856d6ad8eee, entries=150, sequenceid=240, filesize=25.3 K 2023-04-10 02:56:21,814 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/52fd4ad790c04071992ad4289320486e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/52fd4ad790c04071992ad4289320486e 2023-04-10 02:56:21,820 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/52fd4ad790c04071992ad4289320486e, entries=150, sequenceid=240, filesize=11.8 K 2023-04-10 02:56:21,821 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/fc48c85e63a7466da9bd4a66ac6999a9 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/fc48c85e63a7466da9bd4a66ac6999a9 2023-04-10 02:56:21,830 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/fc48c85e63a7466da9bd4a66ac6999a9, entries=150, sequenceid=240, filesize=11.8 K 2023-04-10 02:56:21,831 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=73.80 KB/75570 for 6627f276a3252fdcc4f8b38ace724cf7 in 122ms, sequenceid=240, compaction requested=true 2023-04-10 02:56:21,831 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:21,831 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:21,831 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:21,831 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:21,831 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,831 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:21,831 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:21,833 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,833 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:21,833 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:21,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:21,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=80.51 KB heapSize=225.61 KB 2023-04-10 02:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,835 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:21,835 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:21,835 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:21,835 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:21,835 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:21,835 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78e46f1e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:21,836 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:21,836 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,837 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,838 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:21,838 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:21,838 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:21,838 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,838 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:21,840 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,840 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:21,840 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:21,840 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:21,843 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:21,844 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:21,844 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:21,844 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:21,844 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:21,845 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,845 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:21,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,845 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:21,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,846 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,847 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,853 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,853 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,853 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:21,853 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:21,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:21,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:21,853 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,853 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,853 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,853 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,853 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,853 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:21,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:21,853 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:21,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:21,854 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:21,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:21,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:21,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095441879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:21,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095441879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,954 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:21,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:21,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:21,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:21,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:21,958 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:21,958 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:21,958 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:21,958 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:21,959 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,959 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,961 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:21,961 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:21,962 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,962 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:21,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:21,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:21,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:21,968 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:21,968 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:21,968 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:21,968 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:21,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:21,968 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:21,970 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,970 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:21,972 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:21,972 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:21,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:21,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@139ba45e[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:21,973 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:21,973 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,974 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:21,975 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,975 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:21,976 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:21,979 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:21,979 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:21,979 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:21,980 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:21,980 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:21,983 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,983 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:21,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,984 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:21,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:21,985 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,986 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:21,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:21,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095441984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:21,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095441986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:21,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:21,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:21,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:21,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:21,991 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:21,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:21,992 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:21,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:21,991 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:21,992 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:21,992 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:21,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:22,092 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:22,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:22,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:22,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:22,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:22,104 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:22,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:22,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:22,104 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:22,106 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,106 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,107 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:22,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:22,107 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,108 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,109 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:22,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:22,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:22,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:22,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:22,109 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:22,116 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:22,116 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:22,117 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:22,117 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:22,117 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:22,117 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:22,117 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:22,117 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:22,117 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,117 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:22,117 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:22,119 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,119 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,119 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,119 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:22,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ca5b085[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:22,122 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:22,122 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,124 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:22,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:22,125 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:22,125 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,125 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:22,126 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:22,126 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:22,126 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:22,127 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:22,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:22,131 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:22,131 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:22,131 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:22,131 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:22,131 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:22,133 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,133 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:22,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,133 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,134 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,135 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,137 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:22,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:22,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:22,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:22,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,137 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,138 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:22,138 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:22,138 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:22,138 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:22,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:22,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:22,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:22,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095442190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:22,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095442191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,238 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:22,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:22,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:22,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:22,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:22,244 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:22,244 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:22,244 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:22,244 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:22,244 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,244 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,247 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,247 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:22,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:22,247 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,247 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,248 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410a814c405504240c8b20781303e26ab54 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a814c405504240c8b20781303e26ab54 2023-04-10 02:56:22,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:22,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:22,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:22,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:22,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:22,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:22,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:22,255 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:22,255 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:22,255 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:22,256 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:22,256 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:22,256 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:22,256 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:22,256 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,256 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:22,256 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:22,261 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,261 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,263 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=270, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/a563a792f08640f39ba69cf49032f54c 2023-04-10 02:56:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:22,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a53d462[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:22,266 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:22,266 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,268 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,268 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,268 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:22,275 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:22,278 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:22,278 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:22,278 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:22,278 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:22,278 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:22,283 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,283 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,283 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:22,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:22,284 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,285 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,288 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,288 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,288 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,288 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,288 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,288 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,288 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:22,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:22,289 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:22,289 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:22,288 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:22,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:22,292 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:22,292 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:22,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:22,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:22,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:22,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:22,294 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/39e1f4a482c641989019ff8b7a9079a4 2023-04-10 02:56:22,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:22,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095442311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,389 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:22,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:22,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:22,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:22,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:22,395 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:22,395 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:22,395 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:22,395 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:22,396 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,396 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,397 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,398 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:22,400 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:22,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:22,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:22,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:22,403 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:22,403 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:22,403 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:22,403 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:22,403 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:22,403 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:22,403 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:22,404 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:22,404 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:22,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:22,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:22,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:22,404 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:22,413 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,413 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,413 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,413 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:22,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6194d0a9[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:22,415 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:22,415 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,416 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:22,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:22,416 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:22,416 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,416 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:22,417 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:22,417 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:22,417 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:22,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:22,420 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:22,420 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:22,420 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:22,420 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:22,420 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:22,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:22,423 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:22,423 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:22,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:22,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:22,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,427 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:22,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:22,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:22,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:22,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,428 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:22,428 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:22,428 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:22,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:22,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59034 deadline: 1681095442494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:22,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095442494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,528 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:22,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:22,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:22,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:22,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:22,540 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:22,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:22,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:22,540 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:22,544 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,544 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,546 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:22,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:22,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,546 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:22,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:22,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:22,547 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:22,547 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:22,547 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:22,548 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:22,548 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:22,548 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:22,548 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:22,550 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,550 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fe33191[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:22,555 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:22,555 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,556 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:22,557 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,557 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:22,559 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,559 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,559 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:22,559 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:22,562 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:22,562 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:22,562 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:22,562 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:22,562 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:22,564 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,564 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:22,564 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:22,565 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,567 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:22,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:22,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:22,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:22,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:22,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,570 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,570 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,569 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:22,570 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:22,570 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:22,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:22,670 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:22,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:22,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:22,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:22,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:22,676 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:22,676 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:22,676 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:22,676 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:22,677 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,677 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,678 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:22,678 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,679 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,679 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:22,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:22,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:22,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:22,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:22,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:22,688 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:22,688 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:22,688 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:22,688 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:22,688 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:22,688 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:22,688 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:22,688 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:22,688 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,688 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:22,688 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:22,691 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,692 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e039d70[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:22,693 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:22,693 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,694 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:22,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:22,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:22,695 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,695 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:22,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:22,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:22,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:22,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:22,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:22,700 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:22,700 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:22,700 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:22,701 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:22,701 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:22,702 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,702 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,702 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:22,703 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:22,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:22,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:22,705 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,715 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,715 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:22,715 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:22,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:22,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:22,715 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,715 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,715 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,715 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,715 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:22,715 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,715 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:22,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:22,715 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:22,715 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:22,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:22,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/23c4f6301b874783a53c3716ede242e6 2023-04-10 02:56:22,725 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/a563a792f08640f39ba69cf49032f54c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a563a792f08640f39ba69cf49032f54c 2023-04-10 02:56:22,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a563a792f08640f39ba69cf49032f54c, entries=250, sequenceid=270, filesize=39.2 K 2023-04-10 02:56:22,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/39e1f4a482c641989019ff8b7a9079a4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/39e1f4a482c641989019ff8b7a9079a4 2023-04-10 02:56:22,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/39e1f4a482c641989019ff8b7a9079a4, entries=150, sequenceid=270, filesize=11.9 K 2023-04-10 02:56:22,746 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/23c4f6301b874783a53c3716ede242e6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/23c4f6301b874783a53c3716ede242e6 2023-04-10 02:56:22,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/23c4f6301b874783a53c3716ede242e6, entries=150, sequenceid=270, filesize=11.9 K 2023-04-10 02:56:22,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~245.48 KB/251376, currentSize=93.93 KB/96180 for 6627f276a3252fdcc4f8b38ace724cf7 in 920ms, sequenceid=270, compaction requested=true 2023-04-10 02:56:22,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:22,755 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:22,755 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-10 02:56:22,755 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-10 02:56:22,755 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:22,755 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:22,761 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 221305 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-10 02:56:22,762 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/A is initiating minor compaction (all files) 2023-04-10 02:56:22,762 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/A in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:22,762 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a869565cbedb436b8093ea399e6a8774, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/935e23ce0b3547a387d1ae8272b9d88f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f0da119f606b4fd29843a4568c0daefe, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/c285793ca63146238102dbc7da156144, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3952d3e8be894c9186ea7ebfbbd2aefc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a05d9c164b004ed7ad484e880568e9f1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/e80a79a7ffa6424c88ca3856d6ad8eee, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a563a792f08640f39ba69cf49032f54c] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=216.1 K 2023-04-10 02:56:22,762 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 96341 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-10 02:56:22,762 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/B is initiating minor compaction (all files) 2023-04-10 02:56:22,762 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/B in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:22,763 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/ed46b134a7714c69ba9481cc3dae5f7f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/bd434acadcb14115acbe0c01dee9f6bc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/c0bf5c2122e04a589cd1e3eef2b0ba28, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a4974822bade422d8b24be33eb50b375, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a77a9a6a849a40a6bf4184a55b64db68, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/21edf2a3ef4849cc8122990aece622eb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/52fd4ad790c04071992ad4289320486e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/39e1f4a482c641989019ff8b7a9079a4] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=94.1 K 2023-04-10 02:56:22,763 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting a869565cbedb436b8093ea399e6a8774, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681095377019 2023-04-10 02:56:22,763 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting ed46b134a7714c69ba9481cc3dae5f7f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681095377019 2023-04-10 02:56:22,763 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 935e23ce0b3547a387d1ae8272b9d88f, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681095377751 2023-04-10 02:56:22,763 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting bd434acadcb14115acbe0c01dee9f6bc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681095377751 2023-04-10 02:56:22,764 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting f0da119f606b4fd29843a4568c0daefe, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681095378392 2023-04-10 02:56:22,765 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting c285793ca63146238102dbc7da156144, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1681095379565 2023-04-10 02:56:22,765 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting c0bf5c2122e04a589cd1e3eef2b0ba28, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681095378392 2023-04-10 02:56:22,765 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 3952d3e8be894c9186ea7ebfbbd2aefc, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1681095379885 2023-04-10 02:56:22,765 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting a4974822bade422d8b24be33eb50b375, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1681095379565 2023-04-10 02:56:22,766 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting a77a9a6a849a40a6bf4184a55b64db68, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1681095379885 2023-04-10 02:56:22,767 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting a05d9c164b004ed7ad484e880568e9f1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681095380563 2023-04-10 02:56:22,767 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 21edf2a3ef4849cc8122990aece622eb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681095380563 2023-04-10 02:56:22,767 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting e80a79a7ffa6424c88ca3856d6ad8eee, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681095381171 2023-04-10 02:56:22,767 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 52fd4ad790c04071992ad4289320486e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681095381171 2023-04-10 02:56:22,768 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting a563a792f08640f39ba69cf49032f54c, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681095381823 2023-04-10 02:56:22,768 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 39e1f4a482c641989019ff8b7a9079a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681095381827 2023-04-10 02:56:22,783 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#B#compaction#533 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:22,788 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#A#compaction#532 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:22,816 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:22,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:22,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:22,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:22,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:22,819 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:22,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:22,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:22,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:22,819 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,820 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,821 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:22,821 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:22,821 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:22,821 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:22,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:22,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:22,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:22,822 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:22,822 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:22,822 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:22,824 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:22,824 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:22,824 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:22,824 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:22,824 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-10 02:56:22,824 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:22,824 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:22,824 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:22,824 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:22,825 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:22,825 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:22,860 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041019245400175844649be0d299ea0b8c88 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041019245400175844649be0d299ea0b8c88 2023-04-10 02:56:22,892 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=287, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/1bf08f94f6b44b9c9cc059ad0614c15d 2023-04-10 02:56:23,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:23,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:23,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:23,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095443045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:23,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095443048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:23,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095443151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:23,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095443154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,198 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/d5a8ff2dd90543bb82aed718d9518fd4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/d5a8ff2dd90543bb82aed718d9518fd4 2023-04-10 02:56:23,202 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/B of 6627f276a3252fdcc4f8b38ace724cf7 into d5a8ff2dd90543bb82aed718d9518fd4(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:23,202 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:23,202 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/B, priority=8, startTime=1681095382755; duration=0sec 2023-04-10 02:56:23,202 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:23,202 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-10 02:56:23,205 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 96341 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-10 02:56:23,205 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/C is initiating minor compaction (all files) 2023-04-10 02:56:23,205 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/C in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:23,205 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0029e1d267ce420f8afe97b7a261cd5e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0a65f2b5736d4cedaa62c102b2397507, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a3c140390ce14fdca6ef3ce5e3098d71, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/21f5816f0f684da4a6c4d0ae4fa9bc1a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/492b161d71ca497d8be324fce2c0be9b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/eb3222a0a97d4f31acb213494b13475e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/fc48c85e63a7466da9bd4a66ac6999a9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/23c4f6301b874783a53c3716ede242e6] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=94.1 K 2023-04-10 02:56:23,206 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 0029e1d267ce420f8afe97b7a261cd5e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681095377019 2023-04-10 02:56:23,206 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 0a65f2b5736d4cedaa62c102b2397507, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681095377751 2023-04-10 02:56:23,207 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting a3c140390ce14fdca6ef3ce5e3098d71, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681095378392 2023-04-10 02:56:23,208 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 21f5816f0f684da4a6c4d0ae4fa9bc1a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1681095379565 2023-04-10 02:56:23,208 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 492b161d71ca497d8be324fce2c0be9b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1681095379885 2023-04-10 02:56:23,208 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting eb3222a0a97d4f31acb213494b13475e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681095380563 2023-04-10 02:56:23,209 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting fc48c85e63a7466da9bd4a66ac6999a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681095381171 2023-04-10 02:56:23,209 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 23c4f6301b874783a53c3716ede242e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681095381827 2023-04-10 02:56:23,235 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#C#compaction#536 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:23,244 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/51a979b2cd1d4429a9296729c0a25551 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/51a979b2cd1d4429a9296729c0a25551 2023-04-10 02:56:23,248 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/C of 6627f276a3252fdcc4f8b38ace724cf7 into 51a979b2cd1d4429a9296729c0a25551(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:23,248 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:23,248 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/C, priority=8, startTime=1681095382755; duration=0sec 2023-04-10 02:56:23,248 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:23,322 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/0dbb0338cb224c63a1c45e9e56db2f40 2023-04-10 02:56:23,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:23,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095443322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:23,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095443356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:23,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095443359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,639 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/410898a5179c453b9891be2f6c406997 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/410898a5179c453b9891be2f6c406997 2023-04-10 02:56:23,642 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/A of 6627f276a3252fdcc4f8b38ace724cf7 into 410898a5179c453b9891be2f6c406997(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:23,642 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:23,642 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/A, priority=8, startTime=1681095382754; duration=0sec 2023-04-10 02:56:23,642 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:23,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:23,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095443660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:23,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095443664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,741 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/0b2096f66be64ebfa32d507ca3b76082 2023-04-10 02:56:23,747 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/1bf08f94f6b44b9c9cc059ad0614c15d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/1bf08f94f6b44b9c9cc059ad0614c15d 2023-04-10 02:56:23,751 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/1bf08f94f6b44b9c9cc059ad0614c15d, entries=150, sequenceid=287, filesize=25.4 K 2023-04-10 02:56:23,752 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/0dbb0338cb224c63a1c45e9e56db2f40 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/0dbb0338cb224c63a1c45e9e56db2f40 2023-04-10 02:56:23,755 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/0dbb0338cb224c63a1c45e9e56db2f40, entries=150, sequenceid=287, filesize=11.9 K 2023-04-10 02:56:23,756 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/0b2096f66be64ebfa32d507ca3b76082 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0b2096f66be64ebfa32d507ca3b76082 2023-04-10 02:56:23,759 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0b2096f66be64ebfa32d507ca3b76082, entries=150, sequenceid=287, filesize=11.9 K 2023-04-10 02:56:23,760 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 6627f276a3252fdcc4f8b38ace724cf7 in 936ms, sequenceid=287, compaction requested=false 2023-04-10 02:56:23,760 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:23,760 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:23,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:23,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:23,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:23,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:23,762 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,762 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:23,763 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,763 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:23,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:23,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a57a416[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:23,765 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:23,765 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,768 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,768 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:23,768 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:23,769 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:23,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:23,770 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:23,772 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:23,772 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:23,772 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:23,772 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:23,772 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:23,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:23,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,774 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:23,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:23,776 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:23,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:23,779 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:23,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:23,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:23,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:23,780 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:23,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:23,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:23,779 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,780 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:23,780 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,780 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:23,780 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:23,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:23,880 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:23,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:23,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:23,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:23,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:23,883 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:23,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:23,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:23,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:23,883 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,884 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:23,885 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,885 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:23,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:23,885 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,885 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:23,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:23,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:23,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:23,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:23,887 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:23,887 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:23,887 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-10 02:56:23,887 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:23,887 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:23,887 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:23,887 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:23,887 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:23,887 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:23,898 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410dda2610901034cceabb4f1ecd6d5e32e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410dda2610901034cceabb4f1ecd6d5e32e 2023-04-10 02:56:23,903 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=305, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/7e5f564d37794544ac7c4f514b9c3441 2023-04-10 02:56:23,912 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/6958b1aebb5e4bf990a49549c9b62ec3 2023-04-10 02:56:23,932 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/e469a3b7bdde426f992e870a2d49da3d 2023-04-10 02:56:23,936 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/7e5f564d37794544ac7c4f514b9c3441 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/7e5f564d37794544ac7c4f514b9c3441 2023-04-10 02:56:23,940 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/7e5f564d37794544ac7c4f514b9c3441, entries=150, sequenceid=305, filesize=25.4 K 2023-04-10 02:56:23,941 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/6958b1aebb5e4bf990a49549c9b62ec3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6958b1aebb5e4bf990a49549c9b62ec3 2023-04-10 02:56:23,945 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6958b1aebb5e4bf990a49549c9b62ec3, entries=150, sequenceid=305, filesize=11.9 K 2023-04-10 02:56:23,945 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/e469a3b7bdde426f992e870a2d49da3d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e469a3b7bdde426f992e870a2d49da3d 2023-04-10 02:56:23,949 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e469a3b7bdde426f992e870a2d49da3d, entries=150, sequenceid=305, filesize=11.9 K 2023-04-10 02:56:23,949 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 6627f276a3252fdcc4f8b38ace724cf7 in 62ms, sequenceid=305, compaction requested=true 2023-04-10 02:56:23,949 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:23,949 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:23,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:23,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:23,950 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,951 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:23,951 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:23,953 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,953 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:23,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:23,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:23,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:23,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d81346d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:23,955 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:23,955 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,956 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:23,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:23,957 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,957 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:23,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:23,959 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:23,959 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:23,959 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:23,960 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:23,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:23,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:23,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:23,964 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:23,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:23,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:23,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:23,964 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:23,965 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,965 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,965 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:23,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:23,966 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:23,968 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,971 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:23,971 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,971 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:23,971 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:23,971 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:23,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:23,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:23,971 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:23,972 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:23,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:23,972 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:23,971 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:23,972 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,972 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:23,972 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:23,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:24,072 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:24,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:24,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:24,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:24,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:24,075 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:24,075 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:24,076 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:24,076 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:24,076 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,076 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,078 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:24,078 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:24,078 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,079 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:24,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:24,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:24,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:24,080 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:24,080 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:24,080 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:24,081 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:24,081 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:24,081 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:24,081 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:24,081 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:24,081 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:24,081 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:24,081 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:24,081 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,081 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:24,082 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:24,084 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,084 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,084 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,085 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:24,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:24,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b58e3f5[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:24,086 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:24,086 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,087 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:24,087 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:24,088 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:24,088 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,088 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:24,089 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:24,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:24,092 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:24,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:24,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:24,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:24,092 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:24,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:24,095 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,096 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,101 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:24,101 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:24,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:24,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:24,101 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:24,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:24,101 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:24,101 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,102 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,102 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,102 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,102 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,102 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,102 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:24,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:24,102 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:24,202 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:24,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:24,202 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:56:24,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:24,203 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:24,204 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:24,204 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:24,204 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:24,204 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:24,204 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:24,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:24,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:24,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:24,209 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:24,209 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:24,209 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:24,209 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:24,209 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,209 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:24,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:24,211 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,211 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,211 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:24,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:24,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:24,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:24,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:24,217 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:24,217 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:24,217 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:24,217 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:24,218 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,219 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,219 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,219 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:24,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:24,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49c201f3[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:24,221 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:24,221 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,222 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,222 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:24,222 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:24,222 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:24,222 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,222 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:24,225 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:24,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:24,228 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:24,228 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:24,228 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:24,228 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:24,228 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:24,230 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304104031e33c44404d38a95809294db56402 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304104031e33c44404d38a95809294db56402 2023-04-10 02:56:24,232 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,232 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:24,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,232 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,233 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,234 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,237 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,237 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,237 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,237 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,237 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:24,237 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,237 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,237 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:24,237 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:24,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:24,237 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:24,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095444233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095384228, max timestamp=1681095384236Num uniques 859; 2023-04-10 02:56:24,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095384228, max timestamp=1681095384236Num uniques 859; 2023-04-10 02:56:24,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:24,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:24,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:24,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:24,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:24,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:24,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095384228, max timestamp=1681095384236Num uniques 859; 2023-04-10 02:56:24,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:24,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:24,239 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:24,239 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:24,239 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=317, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/9735a1b07be94dd7976047864271087e 2023-04-10 02:56:24,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:24,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095444296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,339 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:24,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:24,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:24,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095444339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:24,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:24,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:24,347 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:24,347 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:24,347 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:24,347 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:24,347 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,347 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,349 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:24,349 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:24,349 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,349 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:24,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:24,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:24,350 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:24,350 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:24,350 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:24,350 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:24,351 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:24,351 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:24,351 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:24,351 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:24,352 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,353 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,353 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,353 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:24,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec27bc9[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:24,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:24,355 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,357 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,357 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:24,357 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:24,358 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:24,358 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,358 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:24,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:24,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:24,359 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,359 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:24,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:24,362 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:24,362 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:24,362 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:24,362 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:24,362 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:24,363 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,363 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,364 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,364 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:24,365 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:24,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:24,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:24,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:24,367 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:24,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:24,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:24,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,367 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:24,368 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:24,367 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:24,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:24,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:24,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:24,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095444400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,468 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:24,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:24,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:24,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:24,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:24,471 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:24,471 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:24,471 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:24,471 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:24,472 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,472 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,474 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:24,474 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:24,474 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,474 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:24,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:24,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:24,476 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:24,476 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:24,476 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:24,480 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:24,480 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:24,480 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:24,480 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:24,480 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:24,480 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:24,480 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:24,480 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:24,480 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,480 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:24,480 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:24,482 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:24,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ae41220[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:24,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:24,489 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,490 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,490 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:24,490 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,490 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:24,490 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:24,491 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:24,492 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:24,492 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:24,492 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,493 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:24,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:24,495 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:24,495 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:24,495 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:24,495 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:24,495 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:24,497 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,497 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,497 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:24,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,501 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:24,504 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:24,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:24,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:24,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,505 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,505 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:24,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:24,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:24,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,506 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:24,506 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:24,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:24,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:24,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095444545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:24,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59046 deadline: 1681095444595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,600 DEBUG [Thread-3757] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:56:24,606 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:24,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:24,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:24,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095444605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:24,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:24,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:24,610 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:24,610 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:24,610 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:24,610 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:24,610 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,610 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,611 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:24,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:24,612 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,612 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,612 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:24,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:24,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:24,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:24,613 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:24,613 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:24,613 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:24,614 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:24,614 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:24,614 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:24,614 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:24,615 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:24,615 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:24,615 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:24,615 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:24,615 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,615 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:24,615 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:24,618 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,618 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,618 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,618 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:24,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46df09c2[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:24,619 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:24,619 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,625 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,626 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:24,626 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:24,626 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:24,626 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,626 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:24,627 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,627 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,627 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:24,627 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:24,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:24,629 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:24,629 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:24,629 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:24,629 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:24,629 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:24,631 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,631 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:24,631 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:24,632 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,633 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:24,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58984 deadline: 1681095444634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,638 DEBUG [Thread-3755] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8197 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:56:24,639 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:24,639 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:24,639 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:24,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:24,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:24,639 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:24,639 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:24,639 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,639 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:24,639 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,640 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:24,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:24,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:24,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:24,648 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/40ca81c62955481ea1c8e1ee20d8d279 2023-04-10 02:56:24,657 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/58648913cf9d4c679de0bd9b0349a654 2023-04-10 02:56:24,660 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/9735a1b07be94dd7976047864271087e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/9735a1b07be94dd7976047864271087e 2023-04-10 02:56:24,663 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/9735a1b07be94dd7976047864271087e, entries=400, sequenceid=317, filesize=59.8 K 2023-04-10 02:56:24,666 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/40ca81c62955481ea1c8e1ee20d8d279 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/40ca81c62955481ea1c8e1ee20d8d279 2023-04-10 02:56:24,670 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/40ca81c62955481ea1c8e1ee20d8d279, entries=150, sequenceid=317, filesize=11.9 K 2023-04-10 02:56:24,670 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/58648913cf9d4c679de0bd9b0349a654 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/58648913cf9d4c679de0bd9b0349a654 2023-04-10 02:56:24,673 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/58648913cf9d4c679de0bd9b0349a654, entries=150, sequenceid=317, filesize=11.9 K 2023-04-10 02:56:24,674 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6627f276a3252fdcc4f8b38ace724cf7 in 472ms, sequenceid=317, compaction requested=true 2023-04-10 02:56:24,674 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:24,674 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:24,674 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:56:24,674 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:56:24,674 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:24,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:24,675 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 139768 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:56:24,676 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:56:24,676 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/A is initiating minor compaction (all files) 2023-04-10 02:56:24,676 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/B is initiating minor compaction (all files) 2023-04-10 02:56:24,676 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/A in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:24,676 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/B in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:24,676 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/410898a5179c453b9891be2f6c406997, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/1bf08f94f6b44b9c9cc059ad0614c15d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/7e5f564d37794544ac7c4f514b9c3441, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/9735a1b07be94dd7976047864271087e] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=136.5 K 2023-04-10 02:56:24,676 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/d5a8ff2dd90543bb82aed718d9518fd4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/0dbb0338cb224c63a1c45e9e56db2f40, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6958b1aebb5e4bf990a49549c9b62ec3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/40ca81c62955481ea1c8e1ee20d8d279] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=48.0 K 2023-04-10 02:56:24,676 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 410898a5179c453b9891be2f6c406997, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681095381827 2023-04-10 02:56:24,676 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d5a8ff2dd90543bb82aed718d9518fd4, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681095381827 2023-04-10 02:56:24,677 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 1bf08f94f6b44b9c9cc059ad0614c15d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1681095381872 2023-04-10 02:56:24,677 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 0dbb0338cb224c63a1c45e9e56db2f40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1681095381872 2023-04-10 02:56:24,677 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 7e5f564d37794544ac7c4f514b9c3441, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1681095383043 2023-04-10 02:56:24,677 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6958b1aebb5e4bf990a49549c9b62ec3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1681095383043 2023-04-10 02:56:24,677 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 9735a1b07be94dd7976047864271087e, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681095384188 2023-04-10 02:56:24,677 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 40ca81c62955481ea1c8e1ee20d8d279, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681095384198 2023-04-10 02:56:24,685 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#B#compaction#545 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:24,689 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#A#compaction#544 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:24,740 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:24,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:24,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:24,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:24,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:24,743 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:24,743 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:24,743 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:24,743 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:24,743 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,743 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,745 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,745 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:24,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:24,745 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,745 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:24,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:24,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:24,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:24,746 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:24,746 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:24,746 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:24,746 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:24,746 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:24,746 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:24,746 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:24,747 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-10 02:56:24,747 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:24,747 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:24,747 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:24,747 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:24,747 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:24,747 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:24,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:24,850 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:24,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:24,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095444875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:24,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095444908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:24,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:24,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095444979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:25,101 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/b90eb1675622428289806f45b9c752c5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b90eb1675622428289806f45b9c752c5 2023-04-10 02:56:25,106 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/B of 6627f276a3252fdcc4f8b38ace724cf7 into b90eb1675622428289806f45b9c752c5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:25,106 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:25,106 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/B, priority=12, startTime=1681095384674; duration=0sec 2023-04-10 02:56:25,106 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:25,106 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-10 02:56:25,107 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-10 02:56:25,108 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/C is initiating minor compaction (all files) 2023-04-10 02:56:25,108 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/C in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:25,108 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/51a979b2cd1d4429a9296729c0a25551, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0b2096f66be64ebfa32d507ca3b76082, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e469a3b7bdde426f992e870a2d49da3d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/58648913cf9d4c679de0bd9b0349a654] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=48.0 K 2023-04-10 02:56:25,108 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 51a979b2cd1d4429a9296729c0a25551, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681095381827 2023-04-10 02:56:25,109 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 0b2096f66be64ebfa32d507ca3b76082, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1681095381872 2023-04-10 02:56:25,109 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting e469a3b7bdde426f992e870a2d49da3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1681095383043 2023-04-10 02:56:25,109 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 58648913cf9d4c679de0bd9b0349a654, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681095384198 2023-04-10 02:56:25,115 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#C#compaction#547 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:25,163 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041054de7effe3ee4fd2b3df98c78c60c2cf to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041054de7effe3ee4fd2b3df98c78c60c2cf 2023-04-10 02:56:25,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:25,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095445183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:25,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:25,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095445333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:25,339 DEBUG [Thread-3761] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:56:25,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:25,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095445416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:25,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:25,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095445487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:25,533 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/bcfa1e6912c04a1b912302d2d193a1d3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/bcfa1e6912c04a1b912302d2d193a1d3 2023-04-10 02:56:25,535 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/7117bc4af6bf4806aa0b5a24f781ea5b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/7117bc4af6bf4806aa0b5a24f781ea5b 2023-04-10 02:56:25,538 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/A of 6627f276a3252fdcc4f8b38ace724cf7 into bcfa1e6912c04a1b912302d2d193a1d3(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:25,538 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:25,538 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/A, priority=12, startTime=1681095384674; duration=0sec 2023-04-10 02:56:25,538 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:25,540 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/C of 6627f276a3252fdcc4f8b38ace724cf7 into 7117bc4af6bf4806aa0b5a24f781ea5b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:25,540 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:25,540 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/C, priority=12, startTime=1681095384675; duration=0sec 2023-04-10 02:56:25,540 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:25,570 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=353, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/3e433cf29c614b05a971d0ac73aaf83f 2023-04-10 02:56:25,589 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/bd22afd9b98944e4bd3e2335193da3c2 2023-04-10 02:56:25,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:25,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095445995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,014 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/e170e6a9a08b4c8693439a5db79648ef 2023-04-10 02:56:26,017 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/3e433cf29c614b05a971d0ac73aaf83f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3e433cf29c614b05a971d0ac73aaf83f 2023-04-10 02:56:26,020 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3e433cf29c614b05a971d0ac73aaf83f, entries=150, sequenceid=353, filesize=25.4 K 2023-04-10 02:56:26,021 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/bd22afd9b98944e4bd3e2335193da3c2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/bd22afd9b98944e4bd3e2335193da3c2 2023-04-10 02:56:26,023 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/bd22afd9b98944e4bd3e2335193da3c2, entries=150, sequenceid=353, filesize=11.9 K 2023-04-10 02:56:26,024 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/e170e6a9a08b4c8693439a5db79648ef as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e170e6a9a08b4c8693439a5db79648ef 2023-04-10 02:56:26,027 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e170e6a9a08b4c8693439a5db79648ef, entries=150, sequenceid=353, filesize=11.9 K 2023-04-10 02:56:26,028 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 6627f276a3252fdcc4f8b38ace724cf7 in 1281ms, sequenceid=353, compaction requested=false 2023-04-10 02:56:26,028 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:26,028 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:26,028 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:26,028 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:26,028 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,028 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:26,028 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:26,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,030 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:26,030 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,030 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:26,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:26,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fcb4573[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:26,032 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:26,032 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,034 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,034 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:26,034 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:26,034 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:26,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:26,036 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,036 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:26,036 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:26,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:26,036 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:26,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:26,039 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:26,039 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:26,039 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:26,039 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:26,039 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:26,041 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,041 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:26,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,041 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:26,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:26,041 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:26,042 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:26,044 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,044 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:26,044 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:26,044 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,044 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,044 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:26,044 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:26,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:26,044 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,044 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:26,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:26,045 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:26,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:26,044 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:26,045 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:26,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:26,145 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:26,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:26,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:26,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:26,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:26,149 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:26,149 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:26,149 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:26,149 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:26,149 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,149 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:26,151 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:26,151 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:26,151 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,151 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:26,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:26,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:26,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:26,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:26,153 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:26,153 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:26,153 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:26,153 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:26,153 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:26,153 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:26,153 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:26,153 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:56:26,154 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:26,154 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:26,154 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:26,154 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:26,154 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:26,154 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:26,168 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304104c2516da1ff24c8db197adfc23632834 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304104c2516da1ff24c8db197adfc23632834 2023-04-10 02:56:26,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:26,422 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:26,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095386467, max timestamp=1681095386476Num uniques 859; 2023-04-10 02:56:26,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:26,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095386467, max timestamp=1681095386476Num uniques 859; 2023-04-10 02:56:26,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095386467, max timestamp=1681095386476Num uniques 859; 2023-04-10 02:56:26,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:26,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:26,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:26,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095446531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,600 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=367, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/b43e47475a884420b796c440edd206d1 2023-04-10 02:56:26,611 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/a7d20c4ed64844b795824f8e4b9d407e 2023-04-10 02:56:26,622 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/40dfce2ed13d4daf81c4c1d524500f7a 2023-04-10 02:56:26,626 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/b43e47475a884420b796c440edd206d1 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/b43e47475a884420b796c440edd206d1 2023-04-10 02:56:26,631 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/b43e47475a884420b796c440edd206d1, entries=150, sequenceid=367, filesize=25.4 K 2023-04-10 02:56:26,632 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/a7d20c4ed64844b795824f8e4b9d407e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a7d20c4ed64844b795824f8e4b9d407e 2023-04-10 02:56:26,637 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a7d20c4ed64844b795824f8e4b9d407e, entries=150, sequenceid=367, filesize=11.9 K 2023-04-10 02:56:26,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:26,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095446634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,638 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/40dfce2ed13d4daf81c4c1d524500f7a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/40dfce2ed13d4daf81c4c1d524500f7a 2023-04-10 02:56:26,645 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/40dfce2ed13d4daf81c4c1d524500f7a, entries=150, sequenceid=367, filesize=11.9 K 2023-04-10 02:56:26,646 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6627f276a3252fdcc4f8b38ace724cf7 in 493ms, sequenceid=367, compaction requested=true 2023-04-10 02:56:26,646 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:26,646 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:26,649 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,649 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:26,649 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,649 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:26,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:26,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@327a6970[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:26,650 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:26,650 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,651 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,652 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:26,652 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:26,652 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:26,652 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,652 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:26,653 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:26,653 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:26,653 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,653 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:26,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:26,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:26,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:26,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:26,655 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:26,655 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:26,655 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:26,655 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:26,655 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:26,656 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,656 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,657 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:26,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:26,657 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:26,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:26,658 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:26,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:26,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:26,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,662 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:26,662 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:26,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:26,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:26,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:26,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:26,662 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:26,662 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:26,662 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:26,662 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:26,662 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,662 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,662 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:26,662 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:26,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:26,762 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:26,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:26,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:26,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:26,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:26,768 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:26,768 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:26,768 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:26,768 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:26,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:26,771 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,771 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:26,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:26,772 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,772 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:26,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:26,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:26,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:26,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:26,773 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:26,773 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:26,773 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:26,775 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:26,775 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:26,775 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:26,775 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:26,775 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-10 02:56:26,775 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:26,775 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:26,775 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:26,775 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:26,775 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:26,775 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:26,795 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304108da37dcfa132409489ab4a5013d8ce0a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304108da37dcfa132409489ab4a5013d8ce0a 2023-04-10 02:56:26,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:26,843 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:26,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:26,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095446869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:26,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:26,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095446973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:27,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095446998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:27,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095447177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,202 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=403, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/60293474ef284f89b5ab69598fcd64e3 2023-04-10 02:56:27,211 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/9e4d6dfc07c44fe184181f4b8468788f 2023-04-10 02:56:27,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:27,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095447482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,624 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/fabc5eb5061141f3afdd44499e204036 2023-04-10 02:56:27,627 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/60293474ef284f89b5ab69598fcd64e3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/60293474ef284f89b5ab69598fcd64e3 2023-04-10 02:56:27,631 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/60293474ef284f89b5ab69598fcd64e3, entries=150, sequenceid=403, filesize=25.4 K 2023-04-10 02:56:27,632 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/9e4d6dfc07c44fe184181f4b8468788f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/9e4d6dfc07c44fe184181f4b8468788f 2023-04-10 02:56:27,637 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/9e4d6dfc07c44fe184181f4b8468788f, entries=150, sequenceid=403, filesize=11.9 K 2023-04-10 02:56:27,639 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/fabc5eb5061141f3afdd44499e204036 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/fabc5eb5061141f3afdd44499e204036 2023-04-10 02:56:27,641 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/fabc5eb5061141f3afdd44499e204036, entries=150, sequenceid=403, filesize=11.9 K 2023-04-10 02:56:27,642 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 6627f276a3252fdcc4f8b38ace724cf7 in 867ms, sequenceid=403, compaction requested=true 2023-04-10 02:56:27,642 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:27,642 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:27,644 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:27,644 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:27,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:27,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2895180d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:27,646 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:27,646 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:27,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:27,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:27,648 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,648 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:27,650 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:27,653 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:27,653 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:27,653 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:27,653 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:27,653 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:27,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:27,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:27,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:27,661 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:27,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:27,663 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,669 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,669 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:27,669 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:27,669 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:27,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:27,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:27,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:27,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:27,669 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:27,669 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:27,669 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:27,669 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:27,670 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:27,670 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:27,670 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:27,670 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:27,770 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:27,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:27,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:27,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:27,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:27,792 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:27,792 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:27,792 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:27,792 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:27,792 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:27,793 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:27,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:27,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:27,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:27,794 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:27,795 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:27,795 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:27,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:27,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:27,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:27,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:27,800 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:27,800 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:27,800 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:27,800 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:27,800 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:56:27,800 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:27,800 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:27,800 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:27,800 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:27,800 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:27,800 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:27,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:27,990 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:28,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095388028, max timestamp=1681095388056Num uniques 859; 2023-04-10 02:56:28,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:28,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095388028, max timestamp=1681095388056Num uniques 859; 2023-04-10 02:56:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095388028, max timestamp=1681095388056Num uniques 859; 2023-04-10 02:56:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:28,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:28,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095448119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:28,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095448225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,237 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041062c6e0661f2b4b0e83e5be2ddc1a6524 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041062c6e0661f2b4b0e83e5be2ddc1a6524 2023-04-10 02:56:28,249 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=414, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/85668468fc6544e08f1d89d621b460c0 2023-04-10 02:56:28,290 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/afe3a9770b9344a1bad60529fcbb0f5a 2023-04-10 02:56:28,310 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/15246804a6db4cd3b7fdf5af293db746 2023-04-10 02:56:28,314 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/85668468fc6544e08f1d89d621b460c0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/85668468fc6544e08f1d89d621b460c0 2023-04-10 02:56:28,318 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/85668468fc6544e08f1d89d621b460c0, entries=150, sequenceid=414, filesize=25.4 K 2023-04-10 02:56:28,319 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/afe3a9770b9344a1bad60529fcbb0f5a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/afe3a9770b9344a1bad60529fcbb0f5a 2023-04-10 02:56:28,323 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/afe3a9770b9344a1bad60529fcbb0f5a, entries=150, sequenceid=414, filesize=11.9 K 2023-04-10 02:56:28,324 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/15246804a6db4cd3b7fdf5af293db746 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/15246804a6db4cd3b7fdf5af293db746 2023-04-10 02:56:28,328 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/15246804a6db4cd3b7fdf5af293db746, entries=150, sequenceid=414, filesize=11.9 K 2023-04-10 02:56:28,328 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6627f276a3252fdcc4f8b38ace724cf7 in 528ms, sequenceid=414, compaction requested=true 2023-04-10 02:56:28,328 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:28,328 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:28,330 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,330 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:28,330 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,330 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:28,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:28,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@163a95c4[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:28,332 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:28,332 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,333 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,333 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:28,333 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:28,333 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:28,333 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,333 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:28,334 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:28,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:28,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:28,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:28,337 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:28,337 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:28,337 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:28,337 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:28,337 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:28,339 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,339 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:28,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:28,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:28,340 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,341 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:28,345 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:28,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:28,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:28,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:28,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:28,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:28,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:28,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:28,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,345 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,345 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:28,345 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:28,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:28,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:28,432 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-10 02:56:28,432 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:28,432 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:28,432 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:28,432 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:28,432 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:28,432 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:28,446 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:28,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:28,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:28,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:28,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:28,460 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:28,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:28,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:28,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:28,461 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,461 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:28,462 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:28,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:28,462 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:28,463 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410fc7c9f9262c547458506d34799650fa0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fc7c9f9262c547458506d34799650fa0 2023-04-10 02:56:28,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,464 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,464 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:28,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:28,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:28,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:28,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:28,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:28,467 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:28,468 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:28,468 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:28,476 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:28,476 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:28,476 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:28,476 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:28,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095448472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:28,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:28,485 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,485 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:28,486 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,486 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:28,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:28,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@539faaad[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:28,488 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:28,488 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,490 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:28,491 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:28,491 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:28,491 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:28,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:28,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:28,494 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,494 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:28,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:28,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:28,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:28,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:28,498 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:28,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:28,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:28,498 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:28,498 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:28,501 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,501 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:28,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,502 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:28,503 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=451, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/6bd9b74e1cf94564affdc29458c5f86d 2023-04-10 02:56:28,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:28,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:28,504 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:28,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:28,512 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,516 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:28,516 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:28,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:28,516 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:28,516 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:28,516 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:28,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:28,516 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:28,516 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:28,517 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:28,517 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,517 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,517 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:28,517 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:28,538 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/5ae3a96c9e064168ba75dd06ec7136ff 2023-04-10 02:56:28,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/e2a3b85ccaad44c3a3b1e08d3d0239b0 2023-04-10 02:56:28,576 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/6bd9b74e1cf94564affdc29458c5f86d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/6bd9b74e1cf94564affdc29458c5f86d 2023-04-10 02:56:28,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/6bd9b74e1cf94564affdc29458c5f86d, entries=200, sequenceid=451, filesize=32.3 K 2023-04-10 02:56:28,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/5ae3a96c9e064168ba75dd06ec7136ff as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/5ae3a96c9e064168ba75dd06ec7136ff 2023-04-10 02:56:28,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:28,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095448581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,589 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/5ae3a96c9e064168ba75dd06ec7136ff, entries=150, sequenceid=451, filesize=11.9 K 2023-04-10 02:56:28,591 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/e2a3b85ccaad44c3a3b1e08d3d0239b0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e2a3b85ccaad44c3a3b1e08d3d0239b0 2023-04-10 02:56:28,598 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e2a3b85ccaad44c3a3b1e08d3d0239b0, entries=150, sequenceid=451, filesize=11.9 K 2023-04-10 02:56:28,599 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 6627f276a3252fdcc4f8b38ace724cf7 in 168ms, sequenceid=451, compaction requested=true 2023-04-10 02:56:28,599 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:28,599 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:28,599 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-10 02:56:28,599 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-10 02:56:28,601 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:28,601 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:28,606 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163828 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-10 02:56:28,606 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/A is initiating minor compaction (all files) 2023-04-10 02:56:28,606 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/A in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:28,607 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/bcfa1e6912c04a1b912302d2d193a1d3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3e433cf29c614b05a971d0ac73aaf83f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/b43e47475a884420b796c440edd206d1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/60293474ef284f89b5ab69598fcd64e3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/85668468fc6544e08f1d89d621b460c0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/6bd9b74e1cf94564affdc29458c5f86d] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=160.0 K 2023-04-10 02:56:28,607 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73628 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-10 02:56:28,607 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/B is initiating minor compaction (all files) 2023-04-10 02:56:28,607 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/B in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:28,607 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b90eb1675622428289806f45b9c752c5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/bd22afd9b98944e4bd3e2335193da3c2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a7d20c4ed64844b795824f8e4b9d407e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/9e4d6dfc07c44fe184181f4b8468788f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/afe3a9770b9344a1bad60529fcbb0f5a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/5ae3a96c9e064168ba75dd06ec7136ff] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=71.9 K 2023-04-10 02:56:28,608 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting bcfa1e6912c04a1b912302d2d193a1d3, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681095384198 2023-04-10 02:56:28,608 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting b90eb1675622428289806f45b9c752c5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681095384198 2023-04-10 02:56:28,609 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 3e433cf29c614b05a971d0ac73aaf83f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1681095384281 2023-04-10 02:56:28,609 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting bd22afd9b98944e4bd3e2335193da3c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1681095384281 2023-04-10 02:56:28,609 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting a7d20c4ed64844b795824f8e4b9d407e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681095384863 2023-04-10 02:56:28,609 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting b43e47475a884420b796c440edd206d1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681095384863 2023-04-10 02:56:28,609 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 9e4d6dfc07c44fe184181f4b8468788f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681095386514 2023-04-10 02:56:28,610 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting afe3a9770b9344a1bad60529fcbb0f5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681095386860 2023-04-10 02:56:28,611 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 60293474ef284f89b5ab69598fcd64e3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681095386514 2023-04-10 02:56:28,611 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 5ae3a96c9e064168ba75dd06ec7136ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681095388114 2023-04-10 02:56:28,612 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 85668468fc6544e08f1d89d621b460c0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681095386860 2023-04-10 02:56:28,612 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 6bd9b74e1cf94564affdc29458c5f86d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681095388069 2023-04-10 02:56:28,617 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:28,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:28,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:28,638 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#B#compaction#562 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:28,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:28,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:28,641 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:28,641 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:28,641 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:28,641 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:28,642 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,642 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:28,644 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:28,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:28,644 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:28,645 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:28,645 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:28,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:28,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:28,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:28,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:28,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:28,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:28,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:28,659 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#A#compaction#563 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:28,683 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:28,684 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:28,684 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:28,684 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:28,684 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:56:28,684 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:28,684 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:28,684 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:28,684 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:28,684 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:28,689 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:28,713 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/6d507f29a54944b6aa6ab8c41c040ca6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6d507f29a54944b6aa6ab8c41c040ca6 2023-04-10 02:56:28,723 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/B of 6627f276a3252fdcc4f8b38ace724cf7 into 6d507f29a54944b6aa6ab8c41c040ca6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:28,724 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:28,724 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/B, priority=10, startTime=1681095388599; duration=0sec 2023-04-10 02:56:28,724 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:28,724 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-10 02:56:28,726 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73628 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-10 02:56:28,726 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/C is initiating minor compaction (all files) 2023-04-10 02:56:28,727 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/C in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:28,727 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/7117bc4af6bf4806aa0b5a24f781ea5b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e170e6a9a08b4c8693439a5db79648ef, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/40dfce2ed13d4daf81c4c1d524500f7a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/fabc5eb5061141f3afdd44499e204036, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/15246804a6db4cd3b7fdf5af293db746, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e2a3b85ccaad44c3a3b1e08d3d0239b0] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=71.9 K 2023-04-10 02:56:28,727 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 7117bc4af6bf4806aa0b5a24f781ea5b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681095384198 2023-04-10 02:56:28,728 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting e170e6a9a08b4c8693439a5db79648ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1681095384281 2023-04-10 02:56:28,728 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 40dfce2ed13d4daf81c4c1d524500f7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681095384863 2023-04-10 02:56:28,729 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting fabc5eb5061141f3afdd44499e204036, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681095386514 2023-04-10 02:56:28,729 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 15246804a6db4cd3b7fdf5af293db746, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681095386860 2023-04-10 02:56:28,729 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting e2a3b85ccaad44c3a3b1e08d3d0239b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681095388114 2023-04-10 02:56:28,744 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#C#compaction#564 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:28,772 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/af7ad25f5bfb414da2a71ce2c078d291 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/af7ad25f5bfb414da2a71ce2c078d291 2023-04-10 02:56:28,776 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/C of 6627f276a3252fdcc4f8b38ace724cf7 into af7ad25f5bfb414da2a71ce2c078d291(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:28,776 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:28,776 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/C, priority=10, startTime=1681095388601; duration=0sec 2023-04-10 02:56:28,776 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:28,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:28,793 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:28,795 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410cd24ae288f794883ae2190d16e5d6912 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410cd24ae288f794883ae2190d16e5d6912 2023-04-10 02:56:28,808 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=461, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/6d3e309e70624604879bb2cd6d2973c0 2023-04-10 02:56:28,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095388841, max timestamp=1681095388866Num uniques 859; 2023-04-10 02:56:28,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095388841, max timestamp=1681095388866Num uniques 859; 2023-04-10 02:56:28,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:28,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:28,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095388841, max timestamp=1681095388866Num uniques 859; 2023-04-10 02:56:28,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:28,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:28,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095448930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:29,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095449004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,009 DEBUG [Thread-3759] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:56:29,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:29,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095449034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,136 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/2a8716fa5c4c419f8dd1d789d9fecf60 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/2a8716fa5c4c419f8dd1d789d9fecf60 2023-04-10 02:56:29,140 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/A of 6627f276a3252fdcc4f8b38ace724cf7 into 2a8716fa5c4c419f8dd1d789d9fecf60(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:29,140 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:29,140 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/A, priority=10, startTime=1681095388599; duration=0sec 2023-04-10 02:56:29,140 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:29,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:29,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095449238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,244 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/96d0851d1a2343e8bef81e697ae9a42b 2023-04-10 02:56:29,308 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/5c5282adc9464b928c391f041b37a87f 2023-04-10 02:56:29,313 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/6d3e309e70624604879bb2cd6d2973c0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/6d3e309e70624604879bb2cd6d2973c0 2023-04-10 02:56:29,317 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/6d3e309e70624604879bb2cd6d2973c0, entries=150, sequenceid=461, filesize=25.4 K 2023-04-10 02:56:29,318 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/96d0851d1a2343e8bef81e697ae9a42b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/96d0851d1a2343e8bef81e697ae9a42b 2023-04-10 02:56:29,330 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/96d0851d1a2343e8bef81e697ae9a42b, entries=150, sequenceid=461, filesize=11.9 K 2023-04-10 02:56:29,330 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/5c5282adc9464b928c391f041b37a87f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/5c5282adc9464b928c391f041b37a87f 2023-04-10 02:56:29,334 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/5c5282adc9464b928c391f041b37a87f, entries=150, sequenceid=461, filesize=11.9 K 2023-04-10 02:56:29,335 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6627f276a3252fdcc4f8b38ace724cf7 in 651ms, sequenceid=461, compaction requested=false 2023-04-10 02:56:29,335 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:29,335 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:29,335 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:29,336 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:29,336 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,336 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:29,336 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:29,337 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,338 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,338 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,338 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:29,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32cf7847[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:29,339 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:29,339 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:29,341 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:29,341 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,341 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:29,342 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,342 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:29,342 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,342 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:29,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:29,345 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:29,345 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:29,345 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:29,345 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:29,345 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:29,347 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,348 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:29,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,348 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,349 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,351 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,354 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,354 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,354 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,354 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,354 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,354 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,354 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,354 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:29,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:29,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:29,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:29,355 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:29,355 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:29,355 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:29,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:29,355 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:29,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:29,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:29,378 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-10 02:56:29,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:29,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:56:29,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:29,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:56:29,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:29,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-10 02:56:29,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:29,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095449395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,455 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:29,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:29,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:29,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:29,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:29,459 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:29,459 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:29,459 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:29,459 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:29,459 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,459 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,460 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:29,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:29,461 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,461 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:29,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:29,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:29,463 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:29,463 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:29,463 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:29,463 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:29,463 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:29,464 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:29,464 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:29,464 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:29,464 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:29,464 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:29,464 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:29,464 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,464 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:29,464 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:29,470 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:29,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:29,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@321a5b63[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:29,474 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:29,474 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,475 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,475 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:29,475 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:29,475 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:29,475 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,475 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:29,477 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:29,477 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,477 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,477 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:29,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:29,479 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:29,479 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:29,479 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:29,479 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:29,479 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:29,481 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,481 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:29,481 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,482 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,483 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,486 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:29,486 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:29,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:29,486 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,486 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,486 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,486 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,486 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,486 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,487 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:29,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:29,486 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:29,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:29,487 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:29,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:29,487 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:29,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:29,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095449498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:29,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095449544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,587 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:29,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:29,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:29,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:29,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:29,602 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:29,602 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:29,602 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:29,602 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:29,602 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,603 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,608 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:29,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:29,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,608 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:29,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:29,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:29,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:29,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:29,616 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:29,616 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:29,616 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:29,616 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:29,616 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:29,617 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:29,617 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,617 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:29,617 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:29,619 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:29,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@704c6c51[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:29,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:29,622 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,623 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,623 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:29,623 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:29,623 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:29,624 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,624 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:29,629 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:29,635 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:29,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:29,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:29,636 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:29,636 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:29,637 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,637 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:29,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,640 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,641 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,644 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:29,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:29,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:29,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,647 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,648 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,648 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,648 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,648 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,648 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:29,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:29,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:29,648 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:29,648 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:29,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:29,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:29,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095449703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,748 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:29,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:29,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:29,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:29,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:29,754 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:29,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:29,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:29,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:29,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,754 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,756 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:29,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:29,756 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:29,756 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,756 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:29,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:29,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:29,757 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:29,758 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:29,758 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:29,758 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:29,760 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,761 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,761 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e286a37[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:29,762 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:29,762 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,764 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,764 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:29,764 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:29,764 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:29,764 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,764 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:29,766 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,766 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:29,766 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,766 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:29,769 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:29,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:29,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:29,769 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:29,769 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:29,771 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,771 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,771 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,771 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,773 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,776 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:29,776 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,776 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:29,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:29,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:29,776 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:29,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:29,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:29,776 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,776 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,776 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,776 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,776 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,776 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:29,776 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:29,776 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:29,792 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410d9ade8e5ca764b0a9d0c269526e0834f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410d9ade8e5ca764b0a9d0c269526e0834f 2023-04-10 02:56:29,877 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:29,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:29,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:29,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:29,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:29,880 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:29,880 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:29,880 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:29,880 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:29,881 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,881 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,882 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:29,882 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:29,882 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,882 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:29,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:29,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:29,884 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:29,884 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:29,884 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:29,884 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:29,885 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,885 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,886 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,886 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@176c3cf5[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:29,887 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:29,887 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,893 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,893 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:29,893 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:29,893 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,893 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:29,893 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:29,895 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:29,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:29,897 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:29,897 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:29,897 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:29,897 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:29,897 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:29,899 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,899 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,899 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:29,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:29,900 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,901 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:29,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:29,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:29,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:29,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:29,904 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:29,904 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:29,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:29,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:29,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:29,905 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:29,905 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:29,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:29,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:30,005 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:30,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:30,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:30,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:30,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681095450007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:30,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:30,016 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:30,016 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,016 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:30,016 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:30,017 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,017 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,018 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,018 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,019 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,019 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:30,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:30,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:30,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:30,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:30,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:30,024 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,024 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:30,024 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:30,024 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:30,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:30,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:30,024 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,026 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:30,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77cdac38[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:30,029 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:30,029 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,030 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,031 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:30,031 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:30,031 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:30,033 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:30,033 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,033 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,033 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:30,033 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:30,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:30,034 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:30,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:30,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:30,034 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:30,034 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:30,035 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,035 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,036 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,040 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,041 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,048 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,048 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,048 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,048 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,048 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,048 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,048 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,048 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,048 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:30,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:30,048 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,049 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:30,049 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:30,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:30,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095450049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,149 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:30,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:30,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:30,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:30,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:30,152 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:30,152 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,152 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:30,152 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:30,153 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,153 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,154 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,154 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,155 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,155 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:30,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:30,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:30,156 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,156 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:30,156 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:30,156 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:30,159 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,159 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,159 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,159 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:30,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64f07874[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:30,161 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:30,161 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,162 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,162 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:30,162 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:30,162 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:30,162 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,162 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:30,163 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,163 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:30,163 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,163 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:30,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:30,166 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:30,166 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:30,166 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:30,166 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:30,166 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:30,167 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,167 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,167 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,168 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,169 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,172 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,172 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,172 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,172 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,172 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,172 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:30,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:30,172 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,173 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,173 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:30,173 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:30,197 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=502, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/0754de8ffc9a420f8ca6e0d79ec02c01 2023-04-10 02:56:30,206 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/b92724248f9d4973a605dbdec3d41135 2023-04-10 02:56:30,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/f3d3f92f156642cfbe0ccabafdadfdae 2023-04-10 02:56:30,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/0754de8ffc9a420f8ca6e0d79ec02c01 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/0754de8ffc9a420f8ca6e0d79ec02c01 2023-04-10 02:56:30,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/0754de8ffc9a420f8ca6e0d79ec02c01, entries=150, sequenceid=502, filesize=25.4 K 2023-04-10 02:56:30,221 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/b92724248f9d4973a605dbdec3d41135 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b92724248f9d4973a605dbdec3d41135 2023-04-10 02:56:30,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b92724248f9d4973a605dbdec3d41135, entries=150, sequenceid=502, filesize=11.9 K 2023-04-10 02:56:30,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/f3d3f92f156642cfbe0ccabafdadfdae as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f3d3f92f156642cfbe0ccabafdadfdae 2023-04-10 02:56:30,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f3d3f92f156642cfbe0ccabafdadfdae, entries=150, sequenceid=502, filesize=11.9 K 2023-04-10 02:56:30,229 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 6627f276a3252fdcc4f8b38ace724cf7 in 851ms, sequenceid=502, compaction requested=true 2023-04-10 02:56:30,229 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:30,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:30,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:30,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:30,229 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:30,229 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:30,230 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:30,230 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/B is initiating minor compaction (all files) 2023-04-10 02:56:30,230 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/B in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,231 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6d507f29a54944b6aa6ab8c41c040ca6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/96d0851d1a2343e8bef81e697ae9a42b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b92724248f9d4973a605dbdec3d41135] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=36.4 K 2023-04-10 02:56:30,236 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78854 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:30,236 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6d507f29a54944b6aa6ab8c41c040ca6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681095388114 2023-04-10 02:56:30,236 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/A is initiating minor compaction (all files) 2023-04-10 02:56:30,236 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/A in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,236 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/2a8716fa5c4c419f8dd1d789d9fecf60, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/6d3e309e70624604879bb2cd6d2973c0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/0754de8ffc9a420f8ca6e0d79ec02c01] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=77.0 K 2023-04-10 02:56:30,237 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 96d0851d1a2343e8bef81e697ae9a42b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681095388448 2023-04-10 02:56:30,237 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 2a8716fa5c4c419f8dd1d789d9fecf60, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681095388114 2023-04-10 02:56:30,237 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting b92724248f9d4973a605dbdec3d41135, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681095388901 2023-04-10 02:56:30,237 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 6d3e309e70624604879bb2cd6d2973c0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681095388448 2023-04-10 02:56:30,239 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 0754de8ffc9a420f8ca6e0d79ec02c01, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681095388901 2023-04-10 02:56:30,245 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#B#compaction#571 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:30,265 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#A#compaction#572 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:30,273 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:30,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:30,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:30,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:30,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:30,278 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:30,278 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,278 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:30,278 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:30,279 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,279 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,280 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,280 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,281 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,281 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,284 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/2c849c4f266f4f899a6b1def936b9370 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/2c849c4f266f4f899a6b1def936b9370 2023-04-10 02:56:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:30,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:30,288 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:30,288 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,288 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:30,288 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:30,288 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:30,288 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,288 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:30,288 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-10 02:56:30,289 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:30,289 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:30,289 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:30,289 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:30,289 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:30,289 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:30,290 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/A of 6627f276a3252fdcc4f8b38ace724cf7 into 2c849c4f266f4f899a6b1def936b9370(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:30,290 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:30,290 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/A, priority=13, startTime=1681095390229; duration=0sec 2023-04-10 02:56:30,290 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:30,290 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:30,292 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:30,292 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/C is initiating minor compaction (all files) 2023-04-10 02:56:30,292 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/C in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,292 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/af7ad25f5bfb414da2a71ce2c078d291, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/5c5282adc9464b928c391f041b37a87f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f3d3f92f156642cfbe0ccabafdadfdae] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=36.4 K 2023-04-10 02:56:30,292 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting af7ad25f5bfb414da2a71ce2c078d291, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681095388114 2023-04-10 02:56:30,293 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 5c5282adc9464b928c391f041b37a87f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681095388448 2023-04-10 02:56:30,293 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting f3d3f92f156642cfbe0ccabafdadfdae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681095388901 2023-04-10 02:56:30,301 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#C#compaction#574 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:30,311 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410a050e0e1d97d492bb3d528406658b9f9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a050e0e1d97d492bb3d528406658b9f9 2023-04-10 02:56:30,315 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/8892e5b8da9e43c3a9da36adbd942309 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/8892e5b8da9e43c3a9da36adbd942309 2023-04-10 02:56:30,317 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=511, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/8d8e0db037a24c928ed8efd58c2221d3 2023-04-10 02:56:30,321 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/C of 6627f276a3252fdcc4f8b38ace724cf7 into 8892e5b8da9e43c3a9da36adbd942309(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:30,321 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:30,321 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/C, priority=13, startTime=1681095390229; duration=0sec 2023-04-10 02:56:30,321 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:30,333 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/6a5866033f404aff809fb6fa1b9dbf99 2023-04-10 02:56:30,344 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/81a4ee9ec8974b0ea5db97dcb4c5dee5 2023-04-10 02:56:30,349 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/8d8e0db037a24c928ed8efd58c2221d3 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/8d8e0db037a24c928ed8efd58c2221d3 2023-04-10 02:56:30,353 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/8d8e0db037a24c928ed8efd58c2221d3, entries=150, sequenceid=511, filesize=25.4 K 2023-04-10 02:56:30,354 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/6a5866033f404aff809fb6fa1b9dbf99 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6a5866033f404aff809fb6fa1b9dbf99 2023-04-10 02:56:30,359 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6a5866033f404aff809fb6fa1b9dbf99, entries=150, sequenceid=511, filesize=11.9 K 2023-04-10 02:56:30,360 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/81a4ee9ec8974b0ea5db97dcb4c5dee5 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/81a4ee9ec8974b0ea5db97dcb4c5dee5 2023-04-10 02:56:30,363 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/81a4ee9ec8974b0ea5db97dcb4c5dee5, entries=150, sequenceid=511, filesize=11.9 K 2023-04-10 02:56:30,364 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 6627f276a3252fdcc4f8b38ace724cf7 in 76ms, sequenceid=511, compaction requested=false 2023-04-10 02:56:30,364 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:30,365 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,365 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:30,365 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:30,365 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,365 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:30,365 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:30,366 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,366 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,367 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:30,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:30,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7914cf0f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:30,369 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:30,369 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,370 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:30,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:30,370 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:30,370 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,370 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:30,375 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:30,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:30,377 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:30,377 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:30,377 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:30,377 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:30,377 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:30,379 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,379 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,379 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,380 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,381 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,383 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:30,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:30,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:30,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,383 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:30,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,383 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:30,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,484 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:30,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:30,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:30,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:30,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:30,491 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:30,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:30,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:30,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,491 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,493 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,493 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,494 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,494 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:30,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:30,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:30,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:30,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:30,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:30,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:30,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:30,495 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:30,495 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,496 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:30,496 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:30,496 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,511 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:30,511 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:30,511 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,511 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:30,511 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:30,513 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:30,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f69df4c[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:30,515 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:30,515 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,516 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:30,516 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,516 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:30,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:30,516 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:30,518 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,518 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:30,518 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,518 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:30,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:30,521 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:30,521 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:30,521 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:30,521 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:30,521 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:30,524 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,524 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,524 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,525 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,526 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,529 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,529 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,529 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,529 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,529 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,529 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:30,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:30,529 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:30,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,530 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:30,530 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:30,529 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,529 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,530 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:30,552 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-10 02:56:30,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:30,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:30,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:30,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:30,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:30,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:30,563 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041011a767f9be1345a1beb8df91e2b132b5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041011a767f9be1345a1beb8df91e2b132b5 2023-04-10 02:56:30,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095390597, max timestamp=1681095390613Num uniques 859; 2023-04-10 02:56:30,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:30,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095390597, max timestamp=1681095390613Num uniques 859; 2023-04-10 02:56:30,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095390597, max timestamp=1681095390613Num uniques 859; 2023-04-10 02:56:30,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:30,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:30,630 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:30,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:30,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:30,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:30,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:30,635 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:30,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:30,635 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:30,636 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,636 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,644 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,644 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,645 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,645 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:30,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:30,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:30,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:30,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:30,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:30,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:30,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:30,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:30,646 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,646 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:30,646 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:30,646 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:30,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:30,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:30,646 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:30,648 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,648 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,649 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,649 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:30,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:30,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@155f0ef5[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:30,650 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:30,650 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,656 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,657 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:30,657 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,657 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:30,657 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:30,657 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:30,658 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,658 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:30,659 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,659 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:30,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:30,661 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:30,661 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:30,661 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:30,661 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:30,661 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:30,662 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,662 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,662 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,663 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,664 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,666 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,666 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,667 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:30,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:30,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,667 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:30,667 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:30,667 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:30,667 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,667 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,667 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,667 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,667 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,667 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,671 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/811b8c0c0bb24f349ffeca13d89d372c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/811b8c0c0bb24f349ffeca13d89d372c 2023-04-10 02:56:30,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:30,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095450667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,675 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/B of 6627f276a3252fdcc4f8b38ace724cf7 into 811b8c0c0bb24f349ffeca13d89d372c(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:30,675 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:30,675 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/B, priority=13, startTime=1681095390229; duration=0sec 2023-04-10 02:56:30,675 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:30,767 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:30,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:30,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:30,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:30,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:30,771 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:30,771 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,771 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:30,771 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:30,772 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,772 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:30,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095450772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,774 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,774 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,774 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,774 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:30,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:30,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:30,775 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:30,775 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,775 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:30,775 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:30,775 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:30,776 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,776 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:30,776 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:30,776 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:30,777 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,777 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,777 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,778 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:30,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@720be7f2[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:30,779 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:30,779 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,780 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,781 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:30,781 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:30,781 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,781 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:30,781 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:30,782 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,782 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:30,782 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,782 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:30,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:30,785 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:30,785 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:30,785 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:30,785 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:30,785 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:30,787 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,787 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,787 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,788 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,789 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,794 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:30,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:30,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,794 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,795 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,795 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,795 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:30,795 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:30,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,895 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:30,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:30,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:30,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:30,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:30,906 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:30,907 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,907 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:30,907 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:30,907 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,907 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,911 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,911 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,912 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,912 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:30,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:30,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:30,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:30,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:30,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:30,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:30,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:30,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:30,914 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,914 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:30,914 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:30,914 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:30,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:30,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:30,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:30,914 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:30,916 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,916 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,916 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,916 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:30,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2db8320b[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:30,918 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:30,918 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,919 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,919 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:30,919 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:30,919 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,920 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:30,919 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:30,921 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:30,921 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,921 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,921 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:30,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:30,925 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:30,925 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:30,925 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:30,925 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:30,925 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:30,928 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,928 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,928 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:30,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,928 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:30,930 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:30,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:30,933 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:30,932 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:30,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:30,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:30,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:30,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:30,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:30,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:30,933 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:30,933 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:30,933 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:30,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:30,968 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=524, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/9b5a4fd7dacb4478b0433ba39f07780a 2023-04-10 02:56:30,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:30,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095450975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,033 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:31,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:31,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:31,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:31,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:31,038 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:31,038 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:31,038 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:31,038 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:31,038 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,038 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,040 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:31,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:31,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:31,040 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,040 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,041 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:31,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:31,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:31,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:31,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:31,042 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:31,042 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:31,042 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:31,042 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:31,042 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:31,042 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:31,042 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:31,042 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:31,042 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:31,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:31,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:31,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:31,043 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:31,045 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:31,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:31,045 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:31,045 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,046 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:31,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:31,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cea0b4f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:31,047 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:31,047 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,050 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:31,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:31,050 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:31,051 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:31,051 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,051 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:31,051 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:31,052 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:31,052 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:31,052 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:31,052 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:31,052 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:31,052 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:31,052 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:31,054 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,054 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:31,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,055 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:31,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:31,055 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:31,056 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:31,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095451057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,059 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:31,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:31,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:31,059 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:31,059 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:31,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:31,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,059 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,060 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,060 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,060 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,060 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:31,060 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:31,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:31,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:31,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:31,159 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:31,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:31,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:31,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:31,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:31,164 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:31,164 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:31,164 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:31,164 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:31,164 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,164 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,166 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,167 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,167 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:31,168 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:31,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:31,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:31,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:31,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:31,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:31,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:31,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:31,175 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:31,175 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:31,175 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:31,175 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:31,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:31,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:31,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:31,175 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:31,176 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,176 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:31,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:31,177 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,177 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:31,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:31,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f8881[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:31,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:31,178 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,179 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,179 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:31,179 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:31,179 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:31,179 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,179 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:31,180 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,180 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:31,181 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:31,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:31,181 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:31,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:31,183 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:31,183 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:31,183 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:31,183 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:31,183 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:31,185 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,185 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:31,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:31,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,185 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:31,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:31,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:31,185 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:31,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:31,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,186 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:31,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,188 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,189 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:31,189 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,189 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:31,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:31,189 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:31,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:31,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:31,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:31,189 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,189 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,189 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,189 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,189 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,189 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:31,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:31,189 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:31,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:31,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095451278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,290 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:31,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:31,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:31,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:31,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:31,298 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:31,298 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:31,298 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:31,298 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:31,299 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,299 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,300 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:31,300 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:31,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:31,301 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,301 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:31,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:31,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:31,302 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:31,302 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:31,302 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:31,302 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:31,302 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:31,302 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:31,303 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:31,303 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:31,303 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:31,303 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:31,303 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:31,303 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,303 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:31,303 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:31,306 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,306 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:31,306 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,307 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:31,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:31,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:31,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:31,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c36db21[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:31,308 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:31,308 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,310 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,310 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,310 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:31,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:31,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:31,311 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:31,312 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:31,312 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:31,312 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,312 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:31,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:31,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:31,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:31,315 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:31,315 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:31,315 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:31,315 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:31,315 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:31,316 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,316 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,316 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:31,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:31,317 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:31,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,318 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:31,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:31,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:31,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:31,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:31,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:31,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:31,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:31,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:31,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,322 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:31,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:31,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:31,321 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:31,322 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:31,322 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:31,322 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,376 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/31a5158dc7854ec4b058d92eef45bfd7 2023-04-10 02:56:31,388 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/defbeab378b0491e94b2b67da7f11639 2023-04-10 02:56:31,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/9b5a4fd7dacb4478b0433ba39f07780a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/9b5a4fd7dacb4478b0433ba39f07780a 2023-04-10 02:56:31,396 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/9b5a4fd7dacb4478b0433ba39f07780a, entries=150, sequenceid=524, filesize=25.4 K 2023-04-10 02:56:31,397 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/31a5158dc7854ec4b058d92eef45bfd7 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/31a5158dc7854ec4b058d92eef45bfd7 2023-04-10 02:56:31,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/31a5158dc7854ec4b058d92eef45bfd7, entries=150, sequenceid=524, filesize=11.9 K 2023-04-10 02:56:31,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/defbeab378b0491e94b2b67da7f11639 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/defbeab378b0491e94b2b67da7f11639 2023-04-10 02:56:31,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/defbeab378b0491e94b2b67da7f11639, entries=150, sequenceid=524, filesize=11.9 K 2023-04-10 02:56:31,411 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6627f276a3252fdcc4f8b38ace724cf7 in 859ms, sequenceid=524, compaction requested=true 2023-04-10 02:56:31,411 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:31,411 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:31,411 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:31,411 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:31,411 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:31,412 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:31,412 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78956 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:31,412 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/A is initiating minor compaction (all files) 2023-04-10 02:56:31,412 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/A in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:31,413 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/2c849c4f266f4f899a6b1def936b9370, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/8d8e0db037a24c928ed8efd58c2221d3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/9b5a4fd7dacb4478b0433ba39f07780a] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=77.1 K 2023-04-10 02:56:31,413 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:31,413 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/B is initiating minor compaction (all files) 2023-04-10 02:56:31,413 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/B in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:31,413 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/811b8c0c0bb24f349ffeca13d89d372c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6a5866033f404aff809fb6fa1b9dbf99, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/31a5158dc7854ec4b058d92eef45bfd7] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=36.5 K 2023-04-10 02:56:31,413 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 2c849c4f266f4f899a6b1def936b9370, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681095388901 2023-04-10 02:56:31,414 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 811b8c0c0bb24f349ffeca13d89d372c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681095388901 2023-04-10 02:56:31,414 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 8d8e0db037a24c928ed8efd58c2221d3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681095389385 2023-04-10 02:56:31,414 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6a5866033f404aff809fb6fa1b9dbf99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681095389385 2023-04-10 02:56:31,414 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 9b5a4fd7dacb4478b0433ba39f07780a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681095390546 2023-04-10 02:56:31,414 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 31a5158dc7854ec4b058d92eef45bfd7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681095390546 2023-04-10 02:56:31,421 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#B#compaction#580 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:31,422 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:31,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:31,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:31,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:31,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:31,425 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:31,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:31,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:31,425 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:31,426 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,426 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,427 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:31,427 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:31,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:31,428 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#A#compaction#581 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:31,428 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,428 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:31,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:31,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:31,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:31,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:31,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:31,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:31,429 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:31,430 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:31,430 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:31,432 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:31,432 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:31,432 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:31,432 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:31,432 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-10 02:56:31,432 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/496b4192732b44448ae61e819ad6cab0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/496b4192732b44448ae61e819ad6cab0 2023-04-10 02:56:31,432 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:31,433 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:31,433 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:31,433 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:31,433 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:31,433 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:31,439 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/B of 6627f276a3252fdcc4f8b38ace724cf7 into 496b4192732b44448ae61e819ad6cab0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:31,439 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:31,439 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/B, priority=13, startTime=1681095391411; duration=0sec 2023-04-10 02:56:31,440 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:31,440 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:31,440 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:31,441 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/C is initiating minor compaction (all files) 2023-04-10 02:56:31,441 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/C in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:31,441 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/8892e5b8da9e43c3a9da36adbd942309, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/81a4ee9ec8974b0ea5db97dcb4c5dee5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/defbeab378b0491e94b2b67da7f11639] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=36.5 K 2023-04-10 02:56:31,441 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 8892e5b8da9e43c3a9da36adbd942309, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681095388901 2023-04-10 02:56:31,441 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 81a4ee9ec8974b0ea5db97dcb4c5dee5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681095389385 2023-04-10 02:56:31,441 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting defbeab378b0491e94b2b67da7f11639, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681095390546 2023-04-10 02:56:31,456 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#C#compaction#583 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:31,456 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410a1cb7d3d05164802a2de9a1e3f221457 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a1cb7d3d05164802a2de9a1e3f221457 2023-04-10 02:56:31,470 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=562, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/3ac6b883f3a2475e929a6e9eddda13bf 2023-04-10 02:56:31,472 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/ad858859ea894c4d9979643faec88cf7 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/ad858859ea894c4d9979643faec88cf7 2023-04-10 02:56:31,477 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/C of 6627f276a3252fdcc4f8b38ace724cf7 into ad858859ea894c4d9979643faec88cf7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:31,477 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:31,477 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/C, priority=13, startTime=1681095391412; duration=0sec 2023-04-10 02:56:31,477 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:31,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:31,786 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:31,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:31,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095451803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:31,861 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/607496c6cd234554afc75255ad5b2b90 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/607496c6cd234554afc75255ad5b2b90 2023-04-10 02:56:31,866 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/A of 6627f276a3252fdcc4f8b38ace724cf7 into 607496c6cd234554afc75255ad5b2b90(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:31,866 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:31,866 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/A, priority=13, startTime=1681095391411; duration=0sec 2023-04-10 02:56:31,866 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:31,880 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/991c30afb20049d683f68ae9879a3de0 2023-04-10 02:56:31,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:31,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095451907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:32,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095452108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,292 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/f6466eecbc7e487e88e5fea8f6cc3919 2023-04-10 02:56:32,296 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/3ac6b883f3a2475e929a6e9eddda13bf as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3ac6b883f3a2475e929a6e9eddda13bf 2023-04-10 02:56:32,298 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3ac6b883f3a2475e929a6e9eddda13bf, entries=150, sequenceid=562, filesize=25.4 K 2023-04-10 02:56:32,299 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/991c30afb20049d683f68ae9879a3de0 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/991c30afb20049d683f68ae9879a3de0 2023-04-10 02:56:32,303 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/991c30afb20049d683f68ae9879a3de0, entries=150, sequenceid=562, filesize=11.9 K 2023-04-10 02:56:32,303 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/f6466eecbc7e487e88e5fea8f6cc3919 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f6466eecbc7e487e88e5fea8f6cc3919 2023-04-10 02:56:32,307 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f6466eecbc7e487e88e5fea8f6cc3919, entries=150, sequenceid=562, filesize=11.9 K 2023-04-10 02:56:32,308 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 6627f276a3252fdcc4f8b38ace724cf7 in 876ms, sequenceid=562, compaction requested=false 2023-04-10 02:56:32,308 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:32,308 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:32,310 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,310 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:32,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40728cc6[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:32,312 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:32,312 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,313 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,313 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:32,314 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:32,314 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:32,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,314 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:32,316 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,316 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,316 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:32,317 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:32,319 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:32,319 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:32,319 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:32,319 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:32,319 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:32,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,321 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,322 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,324 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,326 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:32,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:32,326 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,326 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,326 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:32,327 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:32,326 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:32,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:32,327 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,327 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,327 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:32,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:32,327 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,327 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:32,327 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:32,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:32,413 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-10 02:56:32,413 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:32,413 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:32,413 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:32,413 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:32,413 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:32,413 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:32,427 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:32,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:32,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:32,430 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410be34a5b86eca495398026765c18a4d48 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410be34a5b86eca495398026765c18a4d48 2023-04-10 02:56:32,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:32,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:32,431 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:32,431 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:32,431 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:32,431 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:32,432 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,432 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,433 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,433 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:32,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:32,433 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,433 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:32,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:32,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:32,435 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:32,435 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:32,435 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:32,441 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:32,441 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:32,441 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:32,441 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:32,444 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,444 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,444 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,444 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:32,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cfe57f4[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:32,445 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:32,445 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,447 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,447 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:32,447 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:32,447 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:32,447 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,447 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:32,448 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,448 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,448 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:32,449 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:32,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:32,451 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:32,451 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:32,451 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:32,451 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:32,451 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:32,453 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,453 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,453 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,454 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,455 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,459 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,459 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,459 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:32,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:32,459 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:32,459 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:32,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:32,459 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:32,459 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,459 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,459 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,460 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,459 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:32,460 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:32,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:32,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:32,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095392446, max timestamp=1681095392469Num uniques 859; 2023-04-10 02:56:32,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:32,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095392446, max timestamp=1681095392469Num uniques 859; 2023-04-10 02:56:32,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:32,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681095392446, max timestamp=1681095392469Num uniques 859; 2023-04-10 02:56:32,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-10 02:56:32,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:32,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095452526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,560 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:32,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:32,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:32,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:32,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:32,562 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:32,563 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:32,563 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:32,563 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:32,563 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,563 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,567 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:32,567 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:32,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:32,567 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,567 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:32,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:32,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:32,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:32,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:32,568 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:32,568 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:32,568 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:32,568 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:32,568 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:32,569 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:32,569 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:32,569 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:32,569 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:32,569 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:32,569 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:32,569 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,569 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:32,569 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,570 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,571 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@168a36a[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:32,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:32,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,574 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:32,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:32,574 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,574 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:32,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:32,576 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,576 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:32,576 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,576 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:32,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:32,578 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:32,578 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:32,578 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:32,579 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:32,579 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:32,580 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,580 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,580 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:32,581 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,582 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:32,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:32,585 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:32,585 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:32,584 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:32,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:32,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:32,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:32,585 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:32,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:32,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:32,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095452631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,685 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:32,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:32,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:32,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:32,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:32,688 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:32,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:32,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:32,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:32,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,688 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,694 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,694 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:32,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:32,694 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,694 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:32,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:32,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:32,695 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:32,695 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:32,696 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:32,696 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:32,696 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:32,696 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:32,696 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,696 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:32,696 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:32,697 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,697 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,698 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,698 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b9d9fb5[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:32,699 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:32,699 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,700 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:32,700 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,700 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:32,701 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:32,701 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:32,701 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,701 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:32,703 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:32,703 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:32,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:32,704 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:32,704 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:32,706 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,706 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,706 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:32,707 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,708 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,710 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:32,710 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:32,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:32,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:32,710 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,710 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,710 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,710 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,711 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,711 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,711 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:32,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:32,710 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:32,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:32,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:32,711 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:32,711 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:32,811 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:32,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:32,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:32,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:32,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:32,814 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:32,814 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:32,814 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:32,815 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:32,815 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,815 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,816 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:32,816 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:32,817 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,817 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:32,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:32,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:32,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:32,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:32,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:32,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:32,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:32,818 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:32,818 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:32,818 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:32,818 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:32,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:32,818 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:32,819 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,819 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:32,819 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:32,820 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,820 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:32,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fdd34bc[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:32,822 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:32,822 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,824 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,824 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:32,824 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:32,824 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,824 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:32,824 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:32,826 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,826 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:32,826 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,826 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:32,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:32,828 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:32,828 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:32,828 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:32,828 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:32,828 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:32,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,830 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:32,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:32,830 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,832 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,846 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=576, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/4e12a14b629b4435bfdd57b6af80a7c6 2023-04-10 02:56:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:32,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095452846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,848 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:32,848 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:32,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:32,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:32,848 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,848 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,848 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,848 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,848 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,848 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:32,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:32,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,849 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:32,849 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:32,848 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,849 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:32,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=576 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/8d5b26cef6f8494fa6ab72f822ae0c93 2023-04-10 02:56:32,949 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:32,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:32,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:32,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:32,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:32,952 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:32,952 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:32,952 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:32,952 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:32,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,953 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,955 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:32,955 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:32,955 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,955 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:32,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:32,956 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:32,956 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:32,956 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:32,956 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:32,956 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:32,957 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:32,957 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:32,957 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:32,957 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:32,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:32,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:32,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:32,957 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:32,958 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,958 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:32,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41928b44[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:32,960 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:32,960 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,964 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:32,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,964 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:32,966 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,966 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:32,966 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,966 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:32,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:32,969 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:32,969 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:32,969 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:32,969 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:32,969 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:32,970 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,970 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,970 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:32,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:32,971 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,972 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,974 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:32,974 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:32,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:32,974 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,974 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:32,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:32,974 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:32,975 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:32,975 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:32,974 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:32,975 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:32,975 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,975 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:32,975 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:32,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:33,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:33,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58992 deadline: 1681095453010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,013 DEBUG [Thread-3759] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:56:33,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:33,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095453070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,073 DEBUG [Thread-3753] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:56:33,075 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:33,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:33,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:33,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:33,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:33,078 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:33,078 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:33,078 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:33,078 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:33,079 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,079 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:33,080 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:33,080 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:33,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:33,080 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,080 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:33,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:33,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:33,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:33,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:33,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:33,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:33,081 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:33,081 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:33,081 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:33,081 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:33,081 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:33,082 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:33,082 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:33,082 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:33,082 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:33,082 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:33,082 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:33,082 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,082 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:33,082 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:33,084 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:33,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:33,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:33,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:33,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:33,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:33,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:33,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a503ef[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:33,087 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:33,087 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,088 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:33,088 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,088 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:33,089 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:33,090 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:33,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:33,092 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:33,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:33,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:33,092 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:33,092 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:33,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,094 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:33,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:33,094 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:33,095 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,099 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:33,099 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:33,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:33,099 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,099 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:33,099 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:33,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:33,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:33,099 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:33,099 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,099 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,099 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,099 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:33,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:33,099 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:33,099 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:33,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:33,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095453150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,200 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:33,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:33,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:33,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:33,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:33,203 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:33,203 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:33,203 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:33,203 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:33,203 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,203 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:33,205 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:33,205 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:33,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:33,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:33,205 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,205 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:33,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:33,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:33,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:33,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:33,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:33,206 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:33,206 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:33,206 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:33,206 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:33,206 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:33,206 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:33,207 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:33,207 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:33,207 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:33,207 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:33,207 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:33,207 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,207 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:33,207 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:33,208 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,208 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:33,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:33,209 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:33,209 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:33,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:33,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@723254d9[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:33,211 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:33,211 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,212 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:33,212 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,212 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:33,215 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:33,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:33,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:33,217 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:33,217 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:33,217 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:33,217 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:33,217 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:33,219 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,219 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,219 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:33,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:33,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:33,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,219 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:33,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:33,221 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:33,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,223 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,223 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:33,223 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:33,223 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,223 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,223 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:33,223 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:33,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:33,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:33,223 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:33,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:33,223 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:33,223 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,223 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:33,223 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:33,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:33,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:33,266 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=576 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/74ad9b58253648f9a15161781bdc9cde 2023-04-10 02:56:33,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/4e12a14b629b4435bfdd57b6af80a7c6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/4e12a14b629b4435bfdd57b6af80a7c6 2023-04-10 02:56:33,272 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/4e12a14b629b4435bfdd57b6af80a7c6, entries=200, sequenceid=576, filesize=32.3 K 2023-04-10 02:56:33,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/8d5b26cef6f8494fa6ab72f822ae0c93 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/8d5b26cef6f8494fa6ab72f822ae0c93 2023-04-10 02:56:33,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/8d5b26cef6f8494fa6ab72f822ae0c93, entries=150, sequenceid=576, filesize=11.9 K 2023-04-10 02:56:33,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/74ad9b58253648f9a15161781bdc9cde as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/74ad9b58253648f9a15161781bdc9cde 2023-04-10 02:56:33,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/74ad9b58253648f9a15161781bdc9cde, entries=150, sequenceid=576, filesize=11.9 K 2023-04-10 02:56:33,279 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6627f276a3252fdcc4f8b38ace724cf7 in 867ms, sequenceid=576, compaction requested=true 2023-04-10 02:56:33,279 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:33,279 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:33,279 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:33,279 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:33,280 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:33,280 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:33,281 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86098 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:33,282 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/A is initiating minor compaction (all files) 2023-04-10 02:56:33,282 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:33,282 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/A in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:33,282 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/B is initiating minor compaction (all files) 2023-04-10 02:56:33,282 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/607496c6cd234554afc75255ad5b2b90, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3ac6b883f3a2475e929a6e9eddda13bf, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/4e12a14b629b4435bfdd57b6af80a7c6] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=84.1 K 2023-04-10 02:56:33,282 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/B in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:33,282 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/496b4192732b44448ae61e819ad6cab0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/991c30afb20049d683f68ae9879a3de0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/8d5b26cef6f8494fa6ab72f822ae0c93] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=36.6 K 2023-04-10 02:56:33,282 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 607496c6cd234554afc75255ad5b2b90, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681095390546 2023-04-10 02:56:33,282 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 496b4192732b44448ae61e819ad6cab0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681095390546 2023-04-10 02:56:33,282 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 3ac6b883f3a2475e929a6e9eddda13bf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681095390643 2023-04-10 02:56:33,282 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 991c30afb20049d683f68ae9879a3de0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681095390643 2023-04-10 02:56:33,283 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 4e12a14b629b4435bfdd57b6af80a7c6, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1681095391788 2023-04-10 02:56:33,283 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 8d5b26cef6f8494fa6ab72f822ae0c93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1681095391796 2023-04-10 02:56:33,288 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#B#compaction#589 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:33,293 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#A#compaction#590 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:33,305 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/6b996595f7824df7aa7c4bf8a92163dc as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6b996595f7824df7aa7c4bf8a92163dc 2023-04-10 02:56:33,318 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/B of 6627f276a3252fdcc4f8b38ace724cf7 into 6b996595f7824df7aa7c4bf8a92163dc(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:33,318 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:33,318 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/B, priority=13, startTime=1681095393279; duration=0sec 2023-04-10 02:56:33,318 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:33,318 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:33,319 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:33,319 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/C is initiating minor compaction (all files) 2023-04-10 02:56:33,319 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/C in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:33,320 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/ad858859ea894c4d9979643faec88cf7, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f6466eecbc7e487e88e5fea8f6cc3919, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/74ad9b58253648f9a15161781bdc9cde] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=36.6 K 2023-04-10 02:56:33,320 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting ad858859ea894c4d9979643faec88cf7, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681095390546 2023-04-10 02:56:33,320 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting f6466eecbc7e487e88e5fea8f6cc3919, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681095390643 2023-04-10 02:56:33,320 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 74ad9b58253648f9a15161781bdc9cde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1681095391796 2023-04-10 02:56:33,323 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:33,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:33,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:33,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:33,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:33,327 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:33,327 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#C#compaction#591 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:33,327 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:33,327 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:33,327 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:33,328 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,328 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:33,329 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:33,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:33,330 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:33,330 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,330 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:33,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:33,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:33,334 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/f5315c3df5e74000aaabd3ded44126f6 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f5315c3df5e74000aaabd3ded44126f6 2023-04-10 02:56:33,337 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/A of 6627f276a3252fdcc4f8b38ace724cf7 into f5315c3df5e74000aaabd3ded44126f6(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:33,337 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:33,337 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/A, priority=13, startTime=1681095393279; duration=0sec 2023-04-10 02:56:33,338 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:33,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:33,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:33,340 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:33,340 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:33,340 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:33,342 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:33,342 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:33,342 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:33,342 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:33,343 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-10 02:56:33,343 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:33,343 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:33,343 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:33,343 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:33,343 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:33,343 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-10 02:56:33,345 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/930c98242ebe4bb79eac67d96848a542 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/930c98242ebe4bb79eac67d96848a542 2023-04-10 02:56:33,354 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/C of 6627f276a3252fdcc4f8b38ace724cf7 into 930c98242ebe4bb79eac67d96848a542(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:33,354 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:33,354 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/C, priority=13, startTime=1681095393280; duration=0sec 2023-04-10 02:56:33,354 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:33,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:33,655 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:33,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:33,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095453679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,760 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410854c6c8a1b674cddacc10cc03c15de05 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410854c6c8a1b674cddacc10cc03c15de05 2023-04-10 02:56:33,764 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=614, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/f7a155a404674be285c1abf7c3cfa66e 2023-04-10 02:56:33,772 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/46e55806829b469092759158e5deaeda 2023-04-10 02:56:33,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:33,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095453782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:33,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:33,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095453985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,182 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/e71f2b41982943e29d4fba3191bf93fa 2023-04-10 02:56:34,185 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/f7a155a404674be285c1abf7c3cfa66e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f7a155a404674be285c1abf7c3cfa66e 2023-04-10 02:56:34,189 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f7a155a404674be285c1abf7c3cfa66e, entries=150, sequenceid=614, filesize=25.4 K 2023-04-10 02:56:34,189 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/46e55806829b469092759158e5deaeda as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/46e55806829b469092759158e5deaeda 2023-04-10 02:56:34,192 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/46e55806829b469092759158e5deaeda, entries=150, sequenceid=614, filesize=11.9 K 2023-04-10 02:56:34,193 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/e71f2b41982943e29d4fba3191bf93fa as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e71f2b41982943e29d4fba3191bf93fa 2023-04-10 02:56:34,196 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e71f2b41982943e29d4fba3191bf93fa, entries=150, sequenceid=614, filesize=11.9 K 2023-04-10 02:56:34,197 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 6627f276a3252fdcc4f8b38ace724cf7 in 855ms, sequenceid=614, compaction requested=false 2023-04-10 02:56:34,197 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:34,197 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:34,197 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:34,197 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:34,197 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,197 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:34,197 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:34,198 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,199 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,199 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,199 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:34,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ece6f7[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:34,201 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:34,201 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,202 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,202 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,202 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:34,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,205 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:34,205 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:34,205 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:34,210 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,210 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:34,210 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,210 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:34,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:34,213 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:34,213 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:34,213 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:34,213 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:34,213 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:34,214 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,215 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:34,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:34,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:34,215 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,216 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,217 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,219 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,219 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,219 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,219 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,219 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,219 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:34,219 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:34,219 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:34,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:34,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:34,219 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:34,220 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:34,220 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:34,220 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:34,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:34,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:34,291 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:56:34,291 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:34,291 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:34,291 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:34,291 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:34,291 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:34,291 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:34,320 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:34,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:34,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:34,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:34,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:34,323 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:34,323 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:34,323 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:34,323 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:34,324 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,324 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,327 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,327 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:34,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:34,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:34,328 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,328 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:34,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:34,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:34,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:34,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:34,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:34,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:34,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:34,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:34,329 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:34,329 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:34,329 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:34,329 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:34,329 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:34,330 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:34,330 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:34,330 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,330 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:34,330 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:34,331 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,331 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,332 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,332 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:34,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f18f85[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:34,333 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:34,333 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,334 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:34,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:34,334 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:34,334 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,335 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:34,335 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,335 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,336 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:34,336 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:34,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:34,338 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:34,338 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:34,338 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:34,338 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:34,338 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:34,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:34,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095454336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,340 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:34,341 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,342 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,343 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,343 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,343 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,343 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,343 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:34,343 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:34,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:34,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:34,343 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:34,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:34,344 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:34,344 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:34,344 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,344 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:34,344 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:34,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:34,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095454440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,444 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:34,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:34,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:34,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:34,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:34,447 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:34,447 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:34,447 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:34,447 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:34,448 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,448 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,449 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:34,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:34,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,449 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:34,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:34,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:34,450 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:34,450 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:34,450 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:34,450 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:34,450 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:34,450 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:34,450 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:34,451 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:34,451 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:34,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:34,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:34,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:34,451 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:34,452 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,452 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,452 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,452 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:34,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:34,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c665e2f[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:34,454 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:34,454 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,456 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,456 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:34,456 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:34,456 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:34,456 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,456 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:34,457 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,457 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:34,457 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,457 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:34,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:34,460 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:34,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:34,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:34,460 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:34,460 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:34,461 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,461 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,461 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,462 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,463 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,465 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,465 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,465 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:34,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:34,465 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:34,465 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:34,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:34,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:34,465 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:34,466 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,466 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:34,466 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:34,466 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,466 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,466 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:34,566 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:34,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:34,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:34,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:34,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:34,569 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:34,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:34,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:34,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:34,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,569 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,572 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:34,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:34,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:34,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,572 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:34,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:34,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:34,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:34,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:34,573 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:34,573 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:34,574 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:34,574 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:34,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:34,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:34,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:34,574 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:34,575 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:34,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c9243e6[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:34,577 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:34,577 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,578 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:34,578 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,578 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:34,579 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:34,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:34,579 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:34,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:34,581 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:34,581 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:34,581 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:34,581 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:34,581 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:34,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,584 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:34,584 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,586 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:34,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:34,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,590 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:34,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:34,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:34,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:34,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:34,590 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:34,591 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:34,591 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:34,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:34,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095454644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:34,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59046 deadline: 1681095454685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,691 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:34,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:34,692 DEBUG [Thread-3757] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18250 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:56:34,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:34,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:34,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58984 deadline: 1681095454691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,704 DEBUG [Thread-3755] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18262 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:56:34,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:34,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:34,705 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:34,705 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:34,705 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:34,705 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:34,705 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,705 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,707 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,707 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:34,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:34,707 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,707 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,709 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410930bfaea024e4949b309c598d3142309 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410930bfaea024e4949b309c598d3142309 2023-04-10 02:56:34,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:34,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:34,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:34,715 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:34,715 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:34,715 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:34,716 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:34,717 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:34,717 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:34,717 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:34,719 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,719 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,723 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:34,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f7adfdd[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:34,727 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:34,727 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,730 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,730 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:34,730 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:34,730 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:34,730 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,730 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:34,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:34,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:34,732 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:34,732 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,736 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=627, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/abb5a2e13528427fa7b318f441da54b4 2023-04-10 02:56:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:34,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:34,740 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:34,740 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:34,740 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:34,740 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:34,740 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:34,743 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,743 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:34,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,743 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:34,744 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:34,745 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,747 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:34,747 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:34,747 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:34,748 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,748 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:34,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:34,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:34,748 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,748 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,748 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,748 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:34,748 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,748 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:34,748 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:34,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:34,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:34,765 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/d7066c492eb346f0a396ea992cdfb64d 2023-04-10 02:56:34,785 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/0bb2c38300774699871b7c2229f0ba4f 2023-04-10 02:56:34,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/abb5a2e13528427fa7b318f441da54b4 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/abb5a2e13528427fa7b318f441da54b4 2023-04-10 02:56:34,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/abb5a2e13528427fa7b318f441da54b4, entries=200, sequenceid=627, filesize=32.3 K 2023-04-10 02:56:34,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/d7066c492eb346f0a396ea992cdfb64d as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/d7066c492eb346f0a396ea992cdfb64d 2023-04-10 02:56:34,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/d7066c492eb346f0a396ea992cdfb64d, entries=150, sequenceid=627, filesize=11.9 K 2023-04-10 02:56:34,797 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/0bb2c38300774699871b7c2229f0ba4f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0bb2c38300774699871b7c2229f0ba4f 2023-04-10 02:56:34,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0bb2c38300774699871b7c2229f0ba4f, entries=150, sequenceid=627, filesize=11.9 K 2023-04-10 02:56:34,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6627f276a3252fdcc4f8b38ace724cf7 in 511ms, sequenceid=627, compaction requested=true 2023-04-10 02:56:34,802 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:34,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:34,803 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:34,803 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:34,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:34,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:34,803 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86200 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:34,804 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:34,804 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/A is initiating minor compaction (all files) 2023-04-10 02:56:34,804 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/B is initiating minor compaction (all files) 2023-04-10 02:56:34,804 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/A in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:34,804 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/B in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:34,804 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f5315c3df5e74000aaabd3ded44126f6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f7a155a404674be285c1abf7c3cfa66e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/abb5a2e13528427fa7b318f441da54b4] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=84.2 K 2023-04-10 02:56:34,804 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6b996595f7824df7aa7c4bf8a92163dc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/46e55806829b469092759158e5deaeda, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/d7066c492eb346f0a396ea992cdfb64d] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=36.7 K 2023-04-10 02:56:34,804 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting f5315c3df5e74000aaabd3ded44126f6, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1681095391796 2023-04-10 02:56:34,805 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6b996595f7824df7aa7c4bf8a92163dc, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1681095391796 2023-04-10 02:56:34,805 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting f7a155a404674be285c1abf7c3cfa66e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1681095392512 2023-04-10 02:56:34,805 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 46e55806829b469092759158e5deaeda, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1681095392512 2023-04-10 02:56:34,805 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting abb5a2e13528427fa7b318f441da54b4, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681095393670 2023-04-10 02:56:34,805 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting d7066c492eb346f0a396ea992cdfb64d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681095393673 2023-04-10 02:56:34,814 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#B#compaction#599 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:34,833 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#A#compaction#598 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:34,848 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:34,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:34,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:34,855 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/2e28bdd0408a4f02a19fc956949c4982 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/2e28bdd0408a4f02a19fc956949c4982 2023-04-10 02:56:34,858 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/B of 6627f276a3252fdcc4f8b38ace724cf7 into 2e28bdd0408a4f02a19fc956949c4982(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:34,858 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:34,858 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/B, priority=13, startTime=1681095394803; duration=0sec 2023-04-10 02:56:34,858 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:34,858 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-10 02:56:34,859 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-10 02:56:34,860 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/C is initiating minor compaction (all files) 2023-04-10 02:56:34,860 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/C in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:34,860 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/930c98242ebe4bb79eac67d96848a542, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e71f2b41982943e29d4fba3191bf93fa, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0bb2c38300774699871b7c2229f0ba4f] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=36.7 K 2023-04-10 02:56:34,860 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 930c98242ebe4bb79eac67d96848a542, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1681095391796 2023-04-10 02:56:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:34,861 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:34,861 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting e71f2b41982943e29d4fba3191bf93fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1681095392512 2023-04-10 02:56:34,861 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:34,861 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:34,861 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:34,862 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,862 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,862 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 0bb2c38300774699871b7c2229f0ba4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681095393673 2023-04-10 02:56:34,864 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:34,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,865 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,866 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:34,866 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:34,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:34,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:34,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:34,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:34,867 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:34,867 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:34,867 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:34,867 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:34,867 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:34,867 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:34,867 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:34,868 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-10 02:56:34,868 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:34,868 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:34,868 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:34,869 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:34,869 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:34,869 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:34,870 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#C#compaction#600 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:34,889 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/a04afb56b7e14b82b17450d7e111791c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a04afb56b7e14b82b17450d7e111791c 2023-04-10 02:56:34,894 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/C of 6627f276a3252fdcc4f8b38ace724cf7 into a04afb56b7e14b82b17450d7e111791c(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:34,894 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:34,894 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/C, priority=13, startTime=1681095394803; duration=0sec 2023-04-10 02:56:34,895 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:34,897 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304105685ce8deb6b40ada18ac456f88bba30 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304105685ce8deb6b40ada18ac456f88bba30 2023-04-10 02:56:34,902 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=648, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/36e8a8955e104823a923f5f15340c018 2023-04-10 02:56:34,925 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/8044bc79c797497ca0f529f4ab9b5bab 2023-04-10 02:56:34,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:34,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:34,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:34,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095454973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:35,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095455076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:35,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095455278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,362 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/f0ba9413bf4e4ad697a4a504bad58748 2023-04-10 02:56:35,366 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/36e8a8955e104823a923f5f15340c018 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/36e8a8955e104823a923f5f15340c018 2023-04-10 02:56:35,370 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/36e8a8955e104823a923f5f15340c018, entries=150, sequenceid=648, filesize=25.4 K 2023-04-10 02:56:35,371 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/8044bc79c797497ca0f529f4ab9b5bab as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/8044bc79c797497ca0f529f4ab9b5bab 2023-04-10 02:56:35,373 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/8044bc79c797497ca0f529f4ab9b5bab, entries=150, sequenceid=648, filesize=11.9 K 2023-04-10 02:56:35,374 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/f0ba9413bf4e4ad697a4a504bad58748 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f0ba9413bf4e4ad697a4a504bad58748 2023-04-10 02:56:35,377 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f0ba9413bf4e4ad697a4a504bad58748, entries=150, sequenceid=648, filesize=11.9 K 2023-04-10 02:56:35,378 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6627f276a3252fdcc4f8b38ace724cf7 in 510ms, sequenceid=648, compaction requested=false 2023-04-10 02:56:35,378 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:35,378 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:35,378 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:35,378 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:35,378 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,378 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:35,378 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:35,380 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:35,380 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:35,380 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:35,380 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:35,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:35,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:35,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:35,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:35,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72faea10[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:35,382 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:35,382 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:35,383 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:35,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:35,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:35,383 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:35,383 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:35,383 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:35,383 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,383 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:35,385 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:35,385 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:35,385 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,385 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:35,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:35,387 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:35,387 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:35,387 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:35,387 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:35,387 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:35,389 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:35,389 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:35,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:35,389 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:35,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:35,390 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:35,390 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:35,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:35,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:35,392 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:35,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:35,392 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:35,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:35,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:35,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:35,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:35,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:35,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:35,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:35,393 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:35,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:35,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:35,393 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:35,393 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:35,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:35,393 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:35,494 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:35,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:35,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:35,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:35,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:35,497 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:35,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:35,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:35,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:35,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:35,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:35,498 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:35,499 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:35,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:35,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:35,499 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,499 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:35,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:35,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:35,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:35,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:35,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:35,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:35,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:35,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:35,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:35,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:35,500 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:35,500 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:35,501 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:35,501 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-10 02:56:35,501 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:35,501 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:35,501 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:35,501 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:35,501 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:35,501 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:35,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:35,585 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:35,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:35,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095455626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,678 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/369a0ccd078b4d6ca452285c63800c7f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/369a0ccd078b4d6ca452285c63800c7f 2023-04-10 02:56:35,684 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/A of 6627f276a3252fdcc4f8b38ace724cf7 into 369a0ccd078b4d6ca452285c63800c7f(size=26.5 K), total size for store is 52.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:35,684 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:35,684 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/A, priority=13, startTime=1681095394803; duration=0sec 2023-04-10 02:56:35,684 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:35,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:35,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38321] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095455730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,927 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304108c5e353455b44c38bd35cdec0667678f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304108c5e353455b44c38bd35cdec0667678f 2023-04-10 02:56:35,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:35,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095455934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:35,942 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=661, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/ea6fef9e472a48b9823ea918a50cf4bf 2023-04-10 02:56:35,961 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/deabff8199bb4de7b9411b090625ca8c 2023-04-10 02:56:35,998 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/bc2587f125064e40b010981326278a2e 2023-04-10 02:56:36,008 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/ea6fef9e472a48b9823ea918a50cf4bf as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ea6fef9e472a48b9823ea918a50cf4bf 2023-04-10 02:56:36,011 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ea6fef9e472a48b9823ea918a50cf4bf, entries=150, sequenceid=661, filesize=25.4 K 2023-04-10 02:56:36,012 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/deabff8199bb4de7b9411b090625ca8c as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/deabff8199bb4de7b9411b090625ca8c 2023-04-10 02:56:36,019 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/deabff8199bb4de7b9411b090625ca8c, entries=150, sequenceid=661, filesize=11.9 K 2023-04-10 02:56:36,019 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/bc2587f125064e40b010981326278a2e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/bc2587f125064e40b010981326278a2e 2023-04-10 02:56:36,026 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/bc2587f125064e40b010981326278a2e, entries=150, sequenceid=661, filesize=11.9 K 2023-04-10 02:56:36,027 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6627f276a3252fdcc4f8b38ace724cf7 in 526ms, sequenceid=661, compaction requested=true 2023-04-10 02:56:36,027 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:36,027 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:36,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:36,027 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:36,029 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:36,029 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:36,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:36,030 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:36,030 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:36,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:36,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:36,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:36,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cf34546[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:36,036 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:36,036 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:36,038 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:36,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:36,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:36,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:36,039 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:36,039 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,039 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:36,041 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:36,041 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:36,041 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:36,041 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:36,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:36,043 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:36,043 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:36,043 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:36,043 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:36,043 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:36,045 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:36,045 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:36,045 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:36,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:36,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:36,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:36,046 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:36,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:36,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:36,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:36,047 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,053 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:36,053 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:36,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:36,053 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,054 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:36,054 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:36,054 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,054 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:36,054 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:36,054 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:36,054 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:36,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:36,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:36,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:36,055 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:36,055 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:36,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:36,155 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:36,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:36,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-10 02:56:36,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-10 02:56:36,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-10 02:56:36,160 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-10 02:56:36,160 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:36,160 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-10 02:56:36,160 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-10 02:56:36,160 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:36,161 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:36,162 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,162 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:36,162 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,162 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-10 02:56:36,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:36,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:36,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-10 02:56:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:36,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-10 02:56:36,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-10 02:56:36,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-10 02:56:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-10 02:56:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-10 02:56:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-10 02:56:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-10 02:56:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-10 02:56:36,168 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:36,168 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. started... 2023-04-10 02:56:36,168 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-10 02:56:36,169 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:36,169 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:36,169 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:36,169 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:36,169 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:36,169 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:36,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:36,242 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. as already flushing 2023-04-10 02:56:36,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:36,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38321] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095456277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,318 DEBUG [Thread-3768] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32bf16cd to 127.0.0.1:60568 2023-04-10 02:56:36,318 DEBUG [Thread-3768] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:36,319 DEBUG [Thread-3764] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b59bdcf to 127.0.0.1:60568 2023-04-10 02:56:36,319 DEBUG [Thread-3764] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:36,322 DEBUG [Thread-3770] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28164190 to 127.0.0.1:60568 2023-04-10 02:56:36,322 DEBUG [Thread-3770] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:36,323 DEBUG [Thread-3772] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x308b3f39 to 127.0.0.1:60568 2023-04-10 02:56:36,323 DEBUG [Thread-3772] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:36,323 DEBUG [Thread-3766] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d765193 to 127.0.0.1:60568 2023-04-10 02:56:36,323 DEBUG [Thread-3766] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:36,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:36,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095456382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:36,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095456585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:36,624 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410bc095e10f1044004975ca9d1676ab9be to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410bc095e10f1044004975ca9d1676ab9be 2023-04-10 02:56:36,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:36,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095456887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,029 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=682, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/ac76f2252b124dbe91db7e88bc49807a 2023-04-10 02:56:37,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:37,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59034 deadline: 1681095457109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,110 DEBUG [Thread-3753] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., hostname=jenkins-hbase4.apache.org,38321,1681095222501, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-10 02:56:37,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-10 02:56:37,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681095457391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,438 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=682 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/fdf3aa887077427a966d4d2615edccce 2023-04-10 02:56:37,454 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=682 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/feeaf874a7994a89acdcb28b32177ec7 2023-04-10 02:56:37,460 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/ac76f2252b124dbe91db7e88bc49807a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ac76f2252b124dbe91db7e88bc49807a 2023-04-10 02:56:37,465 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ac76f2252b124dbe91db7e88bc49807a, entries=150, sequenceid=682, filesize=25.4 K 2023-04-10 02:56:37,466 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/fdf3aa887077427a966d4d2615edccce as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/fdf3aa887077427a966d4d2615edccce 2023-04-10 02:56:37,477 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/fdf3aa887077427a966d4d2615edccce, entries=150, sequenceid=682, filesize=11.9 K 2023-04-10 02:56:37,477 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/feeaf874a7994a89acdcb28b32177ec7 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/feeaf874a7994a89acdcb28b32177ec7 2023-04-10 02:56:37,481 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/feeaf874a7994a89acdcb28b32177ec7, entries=150, sequenceid=682, filesize=11.9 K 2023-04-10 02:56:37,481 INFO [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6627f276a3252fdcc4f8b38ace724cf7 in 1313ms, sequenceid=682, compaction requested=true 2023-04-10 02:56:37,482 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:37,482 DEBUG [rs(jenkins-hbase4.apache.org,38321,1681095222501)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:37,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-10 02:56:37,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-10 02:56:37,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-10 02:56:37,482 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-10 02:56:37,483 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,483 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:37,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:37,484 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-10 02:56:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:37,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:37,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:37,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38321,1681095222501' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-10 02:56:37,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3220b64d[Count = 0] remaining members to acquire global barrier 2023-04-10 02:56:37,486 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-10 02:56:37,486 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:37,487 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:37,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:37,487 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-10 02:56:37,487 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-10 02:56:37,487 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38321,1681095222501' in zk 2023-04-10 02:56:37,487 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,487 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-10 02:56:37,490 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-10 02:56:37,490 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,490 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:37,490 DEBUG [member: 'jenkins-hbase4.apache.org,38321,1681095222501' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-10 02:56:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:37,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:37,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:37,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:37,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38321,1681095222501': 2023-04-10 02:56:37,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38321,1681095222501' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-10 02:56:37,492 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-10 02:56:37,492 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-10 02:56:37,492 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-10 02:56:37,492 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-10 02:56:37,492 INFO [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-10 02:56:37,494 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:37,494 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:37,494 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:37,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-10 02:56:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-10 02:56:37,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-10 02:56:37,494 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-10 02:56:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:37,495 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-10 02:56:37,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-10 02:56:37,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,497 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-10 02:56:37,497 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-10 02:56:37,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-10 02:56:37,497 DEBUG [(jenkins-hbase4.apache.org,36119,1681095221303)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-10 02:56:37,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-10 02:56:37,497 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-10 02:56:37,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-10 02:56:37,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-10 02:56:37,498 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:37,498 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-10 02:56:37,498 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:37,498 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:37,498 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-10 02:56:37,498 DEBUG [Thread-3763] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-10 02:56:37,498 DEBUG [Thread-3763] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-10 02:56:37,498 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-10 02:56:37,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-10 02:56:37,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-10 02:56:37,598 DEBUG [Thread-3763] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-10 02:56:37,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-10 02:56:38,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38321] regionserver.HRegion(9158): Flush requested on 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:38,400 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-10 02:56:38,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:38,400 DEBUG [Thread-3761] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66c25427 to 127.0.0.1:60568 2023-04-10 02:56:38,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:38,400 DEBUG [Thread-3761] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:38,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:38,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:38,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:38,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:38,409 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230410af04d073be844fccaf729d7bbdc64569 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410af04d073be844fccaf729d7bbdc64569 2023-04-10 02:56:38,813 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=695, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/f3b3a832c88546409264b0c59d1401fd 2023-04-10 02:56:38,820 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/6e1b18bf3a334d1c998ef799a7533afc 2023-04-10 02:56:39,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/a9f3ff4eb64945889bdad6c248f9ab5a 2023-04-10 02:56:39,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/f3b3a832c88546409264b0c59d1401fd as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f3b3a832c88546409264b0c59d1401fd 2023-04-10 02:56:39,235 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f3b3a832c88546409264b0c59d1401fd, entries=150, sequenceid=695, filesize=25.4 K 2023-04-10 02:56:39,235 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/6e1b18bf3a334d1c998ef799a7533afc as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6e1b18bf3a334d1c998ef799a7533afc 2023-04-10 02:56:39,239 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6e1b18bf3a334d1c998ef799a7533afc, entries=150, sequenceid=695, filesize=11.9 K 2023-04-10 02:56:39,239 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/a9f3ff4eb64945889bdad6c248f9ab5a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a9f3ff4eb64945889bdad6c248f9ab5a 2023-04-10 02:56:39,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a9f3ff4eb64945889bdad6c248f9ab5a, entries=150, sequenceid=695, filesize=11.9 K 2023-04-10 02:56:39,243 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 6627f276a3252fdcc4f8b38ace724cf7 in 843ms, sequenceid=695, compaction requested=true 2023-04-10 02:56:39,243 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:39,243 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:39,243 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:56:39,243 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:56:39,243 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:39,243 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:39,245 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 131354 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:56:39,245 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62054 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:56:39,245 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/A is initiating minor compaction (all files) 2023-04-10 02:56:39,245 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/B is initiating minor compaction (all files) 2023-04-10 02:56:39,245 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/A in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:39,245 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/B in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:39,245 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/369a0ccd078b4d6ca452285c63800c7f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/36e8a8955e104823a923f5f15340c018, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ea6fef9e472a48b9823ea918a50cf4bf, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ac76f2252b124dbe91db7e88bc49807a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f3b3a832c88546409264b0c59d1401fd] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=128.3 K 2023-04-10 02:56:39,245 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/2e28bdd0408a4f02a19fc956949c4982, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/8044bc79c797497ca0f529f4ab9b5bab, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/deabff8199bb4de7b9411b090625ca8c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/fdf3aa887077427a966d4d2615edccce, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6e1b18bf3a334d1c998ef799a7533afc] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=60.6 K 2023-04-10 02:56:39,245 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 369a0ccd078b4d6ca452285c63800c7f, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681095393673 2023-04-10 02:56:39,246 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 2e28bdd0408a4f02a19fc956949c4982, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681095393673 2023-04-10 02:56:39,246 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting 36e8a8955e104823a923f5f15340c018, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681095394314 2023-04-10 02:56:39,246 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 8044bc79c797497ca0f529f4ab9b5bab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681095394314 2023-04-10 02:56:39,246 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ea6fef9e472a48b9823ea918a50cf4bf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1681095394962 2023-04-10 02:56:39,246 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting deabff8199bb4de7b9411b090625ca8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1681095394962 2023-04-10 02:56:39,246 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting ac76f2252b124dbe91db7e88bc49807a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=682, earliestPutTs=1681095395614 2023-04-10 02:56:39,247 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting fdf3aa887077427a966d4d2615edccce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=682, earliestPutTs=1681095395614 2023-04-10 02:56:39,247 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] compactions.Compactor(207): Compacting f3b3a832c88546409264b0c59d1401fd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1681095396257 2023-04-10 02:56:39,247 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting 6e1b18bf3a334d1c998ef799a7533afc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1681095396257 2023-04-10 02:56:39,256 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#B#compaction#613 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:39,258 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#A#compaction#614 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:39,263 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/32798ea37e8347f7a400a2994c5ebf65 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/32798ea37e8347f7a400a2994c5ebf65 2023-04-10 02:56:39,268 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/B of 6627f276a3252fdcc4f8b38ace724cf7 into 32798ea37e8347f7a400a2994c5ebf65(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:39,268 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:39,268 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/B, priority=11, startTime=1681095399243; duration=0sec 2023-04-10 02:56:39,268 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-10 02:56:39,268 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-10 02:56:39,270 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62054 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-10 02:56:39,270 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1912): 6627f276a3252fdcc4f8b38ace724cf7/C is initiating minor compaction (all files) 2023-04-10 02:56:39,270 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6627f276a3252fdcc4f8b38ace724cf7/C in TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:39,270 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a04afb56b7e14b82b17450d7e111791c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f0ba9413bf4e4ad697a4a504bad58748, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/bc2587f125064e40b010981326278a2e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/feeaf874a7994a89acdcb28b32177ec7, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a9f3ff4eb64945889bdad6c248f9ab5a] into tmpdir=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp, totalSize=60.6 K 2023-04-10 02:56:39,270 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting a04afb56b7e14b82b17450d7e111791c, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681095393673 2023-04-10 02:56:39,270 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting f0ba9413bf4e4ad697a4a504bad58748, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681095394314 2023-04-10 02:56:39,271 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting bc2587f125064e40b010981326278a2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1681095394962 2023-04-10 02:56:39,271 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting feeaf874a7994a89acdcb28b32177ec7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=682, earliestPutTs=1681095395614 2023-04-10 02:56:39,271 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] compactions.Compactor(207): Compacting a9f3ff4eb64945889bdad6c248f9ab5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1681095396257 2023-04-10 02:56:39,272 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/09fddde5fe064f2397aebfc91020a9e2 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/09fddde5fe064f2397aebfc91020a9e2 2023-04-10 02:56:39,275 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/A of 6627f276a3252fdcc4f8b38ace724cf7 into 09fddde5fe064f2397aebfc91020a9e2(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:39,275 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:39,275 INFO [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/A, priority=11, startTime=1681095399243; duration=0sec 2023-04-10 02:56:39,275 DEBUG [RS:0;jenkins-hbase4:38321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:39,278 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] throttle.PressureAwareThroughputController(145): 6627f276a3252fdcc4f8b38ace724cf7#C#compaction#615 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-10 02:56:39,291 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/6f72e08ead46425ebfd289ae42178ddd as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/6f72e08ead46425ebfd289ae42178ddd 2023-04-10 02:56:39,297 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6627f276a3252fdcc4f8b38ace724cf7/C of 6627f276a3252fdcc4f8b38ace724cf7 into 6f72e08ead46425ebfd289ae42178ddd(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-10 02:56:39,297 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:39,297 INFO [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7., storeName=6627f276a3252fdcc4f8b38ace724cf7/C, priority=11, startTime=1681095399243; duration=0sec 2023-04-10 02:56:39,297 DEBUG [RS:0;jenkins-hbase4:38321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-10 02:56:43,057 DEBUG [Thread-3759] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06799455 to 127.0.0.1:60568 2023-04-10 02:56:43,057 DEBUG [Thread-3759] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:44,227 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-10 02:56:44,770 DEBUG [Thread-3755] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4db22237 to 127.0.0.1:60568 2023-04-10 02:56:44,770 DEBUG [Thread-3755] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:44,792 DEBUG [Thread-3757] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08b57aae to 127.0.0.1:60568 2023-04-10 02:56:44,792 DEBUG [Thread-3757] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:47,130 DEBUG [Thread-3753] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e8654 to 127.0.0.1:60568 2023-04-10 02:56:47,130 DEBUG [Thread-3753] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:47,130 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-10 02:56:47,130 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 224 2023-04-10 02:56:47,130 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 16 2023-04-10 02:56:47,130 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 18 2023-04-10 02:56:47,130 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 115 2023-04-10 02:56:47,130 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(394): wrote 206 2023-04-10 02:56:47,130 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-10 02:56:47,130 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 4093 2023-04-10 02:56:47,131 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 4160 2023-04-10 02:56:47,131 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 4053 2023-04-10 02:56:47,131 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 4048 2023-04-10 02:56:47,131 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(398): read 4168 2023-04-10 02:56:47,131 INFO [Listener at localhost/44533] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-10 02:56:47,131 INFO [Listener at localhost/44533] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-10 02:56:47,131 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f06dd50 to 127.0.0.1:60568 2023-04-10 02:56:47,131 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:47,132 INFO [Listener at localhost/44533] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-10 02:56:47,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-10 02:56:47,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:47,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-10 02:56:47,135 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095407135"}]},"ts":"1681095407135"} 2023-04-10 02:56:47,136 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-10 02:56:47,138 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-10 02:56:47,138 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6627f276a3252fdcc4f8b38ace724cf7, UNASSIGN}] 2023-04-10 02:56:47,139 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6627f276a3252fdcc4f8b38ace724cf7, UNASSIGN 2023-04-10 02:56:47,140 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=6627f276a3252fdcc4f8b38ace724cf7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:47,140 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095407140"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681095407140"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681095407140"}]},"ts":"1681095407140"} 2023-04-10 02:56:47,141 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501}] 2023-04-10 02:56:47,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-10 02:56:47,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:47,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6627f276a3252fdcc4f8b38ace724cf7, disabling compactions & flushes 2023-04-10 02:56:47,293 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:47,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:47,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. after waiting 0 ms 2023-04-10 02:56:47,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:47,293 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6627f276a3252fdcc4f8b38ace724cf7 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-10 02:56:47,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=A 2023-04-10 02:56:47,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:47,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=B 2023-04-10 02:56:47,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:47,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6627f276a3252fdcc4f8b38ace724cf7, store=C 2023-04-10 02:56:47,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-10 02:56:47,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-10 02:56:47,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304100d923db42c894497bd41eef050d51266 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304100d923db42c894497bd41eef050d51266 2023-04-10 02:56:47,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-10 02:56:48,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=705, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/cc02ede091324e95afd26a6f3657799b 2023-04-10 02:56:48,116 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=705 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/9c466d9e95814c2fb4142c2afc03b798 2023-04-10 02:56:48,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=705 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/ced234e27bdd4711a4d4ec02a83f6a39 2023-04-10 02:56:48,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/A/cc02ede091324e95afd26a6f3657799b as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/cc02ede091324e95afd26a6f3657799b 2023-04-10 02:56:48,130 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/cc02ede091324e95afd26a6f3657799b, entries=100, sequenceid=705, filesize=18.6 K 2023-04-10 02:56:48,131 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/B/9c466d9e95814c2fb4142c2afc03b798 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/9c466d9e95814c2fb4142c2afc03b798 2023-04-10 02:56:48,134 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/9c466d9e95814c2fb4142c2afc03b798, entries=100, sequenceid=705, filesize=9.5 K 2023-04-10 02:56:48,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/.tmp/C/ced234e27bdd4711a4d4ec02a83f6a39 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/ced234e27bdd4711a4d4ec02a83f6a39 2023-04-10 02:56:48,138 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/ced234e27bdd4711a4d4ec02a83f6a39, entries=100, sequenceid=705, filesize=9.5 K 2023-04-10 02:56:48,138 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 6627f276a3252fdcc4f8b38ace724cf7 in 845ms, sequenceid=705, compaction requested=false 2023-04-10 02:56:48,160 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ba9d3ac58c2f4b7bb923548686281c4d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/b717eb6b935c448b96519483def42a9c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/dbc258adb7cb423eb114c6c3ec3005af, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a869565cbedb436b8093ea399e6a8774, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/935e23ce0b3547a387d1ae8272b9d88f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f0da119f606b4fd29843a4568c0daefe, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/c285793ca63146238102dbc7da156144, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3952d3e8be894c9186ea7ebfbbd2aefc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a05d9c164b004ed7ad484e880568e9f1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/e80a79a7ffa6424c88ca3856d6ad8eee, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a563a792f08640f39ba69cf49032f54c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/410898a5179c453b9891be2f6c406997, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/1bf08f94f6b44b9c9cc059ad0614c15d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/7e5f564d37794544ac7c4f514b9c3441, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/9735a1b07be94dd7976047864271087e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/bcfa1e6912c04a1b912302d2d193a1d3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3e433cf29c614b05a971d0ac73aaf83f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/b43e47475a884420b796c440edd206d1, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/60293474ef284f89b5ab69598fcd64e3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/85668468fc6544e08f1d89d621b460c0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/6bd9b74e1cf94564affdc29458c5f86d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/2a8716fa5c4c419f8dd1d789d9fecf60, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/6d3e309e70624604879bb2cd6d2973c0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/2c849c4f266f4f899a6b1def936b9370, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/0754de8ffc9a420f8ca6e0d79ec02c01, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/8d8e0db037a24c928ed8efd58c2221d3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/607496c6cd234554afc75255ad5b2b90, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/9b5a4fd7dacb4478b0433ba39f07780a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3ac6b883f3a2475e929a6e9eddda13bf, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/4e12a14b629b4435bfdd57b6af80a7c6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f5315c3df5e74000aaabd3ded44126f6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f7a155a404674be285c1abf7c3cfa66e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/abb5a2e13528427fa7b318f441da54b4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/369a0ccd078b4d6ca452285c63800c7f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/36e8a8955e104823a923f5f15340c018, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ea6fef9e472a48b9823ea918a50cf4bf, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ac76f2252b124dbe91db7e88bc49807a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f3b3a832c88546409264b0c59d1401fd] to archive 2023-04-10 02:56:48,161 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:56:48,162 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ba9d3ac58c2f4b7bb923548686281c4d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ba9d3ac58c2f4b7bb923548686281c4d 2023-04-10 02:56:48,163 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/b717eb6b935c448b96519483def42a9c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/b717eb6b935c448b96519483def42a9c 2023-04-10 02:56:48,164 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/dbc258adb7cb423eb114c6c3ec3005af to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/dbc258adb7cb423eb114c6c3ec3005af 2023-04-10 02:56:48,165 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a869565cbedb436b8093ea399e6a8774 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a869565cbedb436b8093ea399e6a8774 2023-04-10 02:56:48,166 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/935e23ce0b3547a387d1ae8272b9d88f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/935e23ce0b3547a387d1ae8272b9d88f 2023-04-10 02:56:48,166 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f0da119f606b4fd29843a4568c0daefe to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f0da119f606b4fd29843a4568c0daefe 2023-04-10 02:56:48,167 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/c285793ca63146238102dbc7da156144 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/c285793ca63146238102dbc7da156144 2023-04-10 02:56:48,168 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3952d3e8be894c9186ea7ebfbbd2aefc to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3952d3e8be894c9186ea7ebfbbd2aefc 2023-04-10 02:56:48,168 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a05d9c164b004ed7ad484e880568e9f1 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a05d9c164b004ed7ad484e880568e9f1 2023-04-10 02:56:48,169 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/e80a79a7ffa6424c88ca3856d6ad8eee to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/e80a79a7ffa6424c88ca3856d6ad8eee 2023-04-10 02:56:48,170 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a563a792f08640f39ba69cf49032f54c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/a563a792f08640f39ba69cf49032f54c 2023-04-10 02:56:48,171 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/410898a5179c453b9891be2f6c406997 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/410898a5179c453b9891be2f6c406997 2023-04-10 02:56:48,171 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/1bf08f94f6b44b9c9cc059ad0614c15d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/1bf08f94f6b44b9c9cc059ad0614c15d 2023-04-10 02:56:48,172 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/7e5f564d37794544ac7c4f514b9c3441 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/7e5f564d37794544ac7c4f514b9c3441 2023-04-10 02:56:48,173 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/9735a1b07be94dd7976047864271087e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/9735a1b07be94dd7976047864271087e 2023-04-10 02:56:48,174 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/bcfa1e6912c04a1b912302d2d193a1d3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/bcfa1e6912c04a1b912302d2d193a1d3 2023-04-10 02:56:48,174 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3e433cf29c614b05a971d0ac73aaf83f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3e433cf29c614b05a971d0ac73aaf83f 2023-04-10 02:56:48,175 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/b43e47475a884420b796c440edd206d1 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/b43e47475a884420b796c440edd206d1 2023-04-10 02:56:48,176 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/60293474ef284f89b5ab69598fcd64e3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/60293474ef284f89b5ab69598fcd64e3 2023-04-10 02:56:48,177 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/85668468fc6544e08f1d89d621b460c0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/85668468fc6544e08f1d89d621b460c0 2023-04-10 02:56:48,177 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/6bd9b74e1cf94564affdc29458c5f86d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/6bd9b74e1cf94564affdc29458c5f86d 2023-04-10 02:56:48,178 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/2a8716fa5c4c419f8dd1d789d9fecf60 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/2a8716fa5c4c419f8dd1d789d9fecf60 2023-04-10 02:56:48,179 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/6d3e309e70624604879bb2cd6d2973c0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/6d3e309e70624604879bb2cd6d2973c0 2023-04-10 02:56:48,180 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/2c849c4f266f4f899a6b1def936b9370 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/2c849c4f266f4f899a6b1def936b9370 2023-04-10 02:56:48,180 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/0754de8ffc9a420f8ca6e0d79ec02c01 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/0754de8ffc9a420f8ca6e0d79ec02c01 2023-04-10 02:56:48,181 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/8d8e0db037a24c928ed8efd58c2221d3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/8d8e0db037a24c928ed8efd58c2221d3 2023-04-10 02:56:48,182 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/607496c6cd234554afc75255ad5b2b90 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/607496c6cd234554afc75255ad5b2b90 2023-04-10 02:56:48,183 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/9b5a4fd7dacb4478b0433ba39f07780a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/9b5a4fd7dacb4478b0433ba39f07780a 2023-04-10 02:56:48,183 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3ac6b883f3a2475e929a6e9eddda13bf to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/3ac6b883f3a2475e929a6e9eddda13bf 2023-04-10 02:56:48,184 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/4e12a14b629b4435bfdd57b6af80a7c6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/4e12a14b629b4435bfdd57b6af80a7c6 2023-04-10 02:56:48,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f5315c3df5e74000aaabd3ded44126f6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f5315c3df5e74000aaabd3ded44126f6 2023-04-10 02:56:48,186 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f7a155a404674be285c1abf7c3cfa66e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f7a155a404674be285c1abf7c3cfa66e 2023-04-10 02:56:48,187 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/abb5a2e13528427fa7b318f441da54b4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/abb5a2e13528427fa7b318f441da54b4 2023-04-10 02:56:48,187 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/369a0ccd078b4d6ca452285c63800c7f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/369a0ccd078b4d6ca452285c63800c7f 2023-04-10 02:56:48,188 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/36e8a8955e104823a923f5f15340c018 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/36e8a8955e104823a923f5f15340c018 2023-04-10 02:56:48,189 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ea6fef9e472a48b9823ea918a50cf4bf to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ea6fef9e472a48b9823ea918a50cf4bf 2023-04-10 02:56:48,190 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ac76f2252b124dbe91db7e88bc49807a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/ac76f2252b124dbe91db7e88bc49807a 2023-04-10 02:56:48,190 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f3b3a832c88546409264b0c59d1401fd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/f3b3a832c88546409264b0c59d1401fd 2023-04-10 02:56:48,193 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/3cc63c4c175f413bbd06d9243099b29a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b12e04768bce4f6e825ad839293be42d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/ed46b134a7714c69ba9481cc3dae5f7f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/7d171740fec54061b76b18bb8f4407d8, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/bd434acadcb14115acbe0c01dee9f6bc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/c0bf5c2122e04a589cd1e3eef2b0ba28, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a4974822bade422d8b24be33eb50b375, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a77a9a6a849a40a6bf4184a55b64db68, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/21edf2a3ef4849cc8122990aece622eb, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/52fd4ad790c04071992ad4289320486e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/d5a8ff2dd90543bb82aed718d9518fd4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/39e1f4a482c641989019ff8b7a9079a4, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/0dbb0338cb224c63a1c45e9e56db2f40, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6958b1aebb5e4bf990a49549c9b62ec3, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b90eb1675622428289806f45b9c752c5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/40ca81c62955481ea1c8e1ee20d8d279, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/bd22afd9b98944e4bd3e2335193da3c2, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a7d20c4ed64844b795824f8e4b9d407e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/9e4d6dfc07c44fe184181f4b8468788f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/afe3a9770b9344a1bad60529fcbb0f5a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6d507f29a54944b6aa6ab8c41c040ca6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/5ae3a96c9e064168ba75dd06ec7136ff, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/96d0851d1a2343e8bef81e697ae9a42b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/811b8c0c0bb24f349ffeca13d89d372c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b92724248f9d4973a605dbdec3d41135, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6a5866033f404aff809fb6fa1b9dbf99, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/496b4192732b44448ae61e819ad6cab0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/31a5158dc7854ec4b058d92eef45bfd7, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/991c30afb20049d683f68ae9879a3de0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6b996595f7824df7aa7c4bf8a92163dc, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/8d5b26cef6f8494fa6ab72f822ae0c93, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/46e55806829b469092759158e5deaeda, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/2e28bdd0408a4f02a19fc956949c4982, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/d7066c492eb346f0a396ea992cdfb64d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/8044bc79c797497ca0f529f4ab9b5bab, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/deabff8199bb4de7b9411b090625ca8c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/fdf3aa887077427a966d4d2615edccce, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6e1b18bf3a334d1c998ef799a7533afc] to archive 2023-04-10 02:56:48,194 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:56:48,195 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/3cc63c4c175f413bbd06d9243099b29a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/3cc63c4c175f413bbd06d9243099b29a 2023-04-10 02:56:48,196 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b12e04768bce4f6e825ad839293be42d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b12e04768bce4f6e825ad839293be42d 2023-04-10 02:56:48,196 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/ed46b134a7714c69ba9481cc3dae5f7f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/ed46b134a7714c69ba9481cc3dae5f7f 2023-04-10 02:56:48,197 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/7d171740fec54061b76b18bb8f4407d8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/7d171740fec54061b76b18bb8f4407d8 2023-04-10 02:56:48,198 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/bd434acadcb14115acbe0c01dee9f6bc to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/bd434acadcb14115acbe0c01dee9f6bc 2023-04-10 02:56:48,198 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/c0bf5c2122e04a589cd1e3eef2b0ba28 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/c0bf5c2122e04a589cd1e3eef2b0ba28 2023-04-10 02:56:48,199 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a4974822bade422d8b24be33eb50b375 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a4974822bade422d8b24be33eb50b375 2023-04-10 02:56:48,199 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a77a9a6a849a40a6bf4184a55b64db68 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a77a9a6a849a40a6bf4184a55b64db68 2023-04-10 02:56:48,200 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/21edf2a3ef4849cc8122990aece622eb to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/21edf2a3ef4849cc8122990aece622eb 2023-04-10 02:56:48,201 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/52fd4ad790c04071992ad4289320486e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/52fd4ad790c04071992ad4289320486e 2023-04-10 02:56:48,201 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/d5a8ff2dd90543bb82aed718d9518fd4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/d5a8ff2dd90543bb82aed718d9518fd4 2023-04-10 02:56:48,202 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/39e1f4a482c641989019ff8b7a9079a4 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/39e1f4a482c641989019ff8b7a9079a4 2023-04-10 02:56:48,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/0dbb0338cb224c63a1c45e9e56db2f40 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/0dbb0338cb224c63a1c45e9e56db2f40 2023-04-10 02:56:48,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6958b1aebb5e4bf990a49549c9b62ec3 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6958b1aebb5e4bf990a49549c9b62ec3 2023-04-10 02:56:48,204 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b90eb1675622428289806f45b9c752c5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b90eb1675622428289806f45b9c752c5 2023-04-10 02:56:48,205 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/40ca81c62955481ea1c8e1ee20d8d279 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/40ca81c62955481ea1c8e1ee20d8d279 2023-04-10 02:56:48,205 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/bd22afd9b98944e4bd3e2335193da3c2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/bd22afd9b98944e4bd3e2335193da3c2 2023-04-10 02:56:48,206 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a7d20c4ed64844b795824f8e4b9d407e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/a7d20c4ed64844b795824f8e4b9d407e 2023-04-10 02:56:48,207 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/9e4d6dfc07c44fe184181f4b8468788f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/9e4d6dfc07c44fe184181f4b8468788f 2023-04-10 02:56:48,207 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/afe3a9770b9344a1bad60529fcbb0f5a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/afe3a9770b9344a1bad60529fcbb0f5a 2023-04-10 02:56:48,208 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6d507f29a54944b6aa6ab8c41c040ca6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6d507f29a54944b6aa6ab8c41c040ca6 2023-04-10 02:56:48,209 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/5ae3a96c9e064168ba75dd06ec7136ff to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/5ae3a96c9e064168ba75dd06ec7136ff 2023-04-10 02:56:48,212 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/96d0851d1a2343e8bef81e697ae9a42b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/96d0851d1a2343e8bef81e697ae9a42b 2023-04-10 02:56:48,213 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/811b8c0c0bb24f349ffeca13d89d372c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/811b8c0c0bb24f349ffeca13d89d372c 2023-04-10 02:56:48,214 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b92724248f9d4973a605dbdec3d41135 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/b92724248f9d4973a605dbdec3d41135 2023-04-10 02:56:48,215 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6a5866033f404aff809fb6fa1b9dbf99 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6a5866033f404aff809fb6fa1b9dbf99 2023-04-10 02:56:48,216 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/496b4192732b44448ae61e819ad6cab0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/496b4192732b44448ae61e819ad6cab0 2023-04-10 02:56:48,217 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/31a5158dc7854ec4b058d92eef45bfd7 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/31a5158dc7854ec4b058d92eef45bfd7 2023-04-10 02:56:48,218 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/991c30afb20049d683f68ae9879a3de0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/991c30afb20049d683f68ae9879a3de0 2023-04-10 02:56:48,219 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6b996595f7824df7aa7c4bf8a92163dc to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6b996595f7824df7aa7c4bf8a92163dc 2023-04-10 02:56:48,220 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/8d5b26cef6f8494fa6ab72f822ae0c93 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/8d5b26cef6f8494fa6ab72f822ae0c93 2023-04-10 02:56:48,221 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/46e55806829b469092759158e5deaeda to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/46e55806829b469092759158e5deaeda 2023-04-10 02:56:48,222 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/2e28bdd0408a4f02a19fc956949c4982 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/2e28bdd0408a4f02a19fc956949c4982 2023-04-10 02:56:48,223 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/d7066c492eb346f0a396ea992cdfb64d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/d7066c492eb346f0a396ea992cdfb64d 2023-04-10 02:56:48,223 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/8044bc79c797497ca0f529f4ab9b5bab to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/8044bc79c797497ca0f529f4ab9b5bab 2023-04-10 02:56:48,224 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/deabff8199bb4de7b9411b090625ca8c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/deabff8199bb4de7b9411b090625ca8c 2023-04-10 02:56:48,225 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/fdf3aa887077427a966d4d2615edccce to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/fdf3aa887077427a966d4d2615edccce 2023-04-10 02:56:48,226 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6e1b18bf3a334d1c998ef799a7533afc to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/6e1b18bf3a334d1c998ef799a7533afc 2023-04-10 02:56:48,229 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/b9c48bd3b53c4fcba79b854a519a0942, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/9bf36dcb425846df93fc0df4ad232626, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0029e1d267ce420f8afe97b7a261cd5e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/143aa7833e6e4a34aeffa274ff702a46, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0a65f2b5736d4cedaa62c102b2397507, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a3c140390ce14fdca6ef3ce5e3098d71, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/21f5816f0f684da4a6c4d0ae4fa9bc1a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/492b161d71ca497d8be324fce2c0be9b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/eb3222a0a97d4f31acb213494b13475e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/fc48c85e63a7466da9bd4a66ac6999a9, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/51a979b2cd1d4429a9296729c0a25551, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/23c4f6301b874783a53c3716ede242e6, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0b2096f66be64ebfa32d507ca3b76082, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e469a3b7bdde426f992e870a2d49da3d, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/7117bc4af6bf4806aa0b5a24f781ea5b, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/58648913cf9d4c679de0bd9b0349a654, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e170e6a9a08b4c8693439a5db79648ef, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/40dfce2ed13d4daf81c4c1d524500f7a, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/fabc5eb5061141f3afdd44499e204036, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/15246804a6db4cd3b7fdf5af293db746, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/af7ad25f5bfb414da2a71ce2c078d291, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e2a3b85ccaad44c3a3b1e08d3d0239b0, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/5c5282adc9464b928c391f041b37a87f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/8892e5b8da9e43c3a9da36adbd942309, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f3d3f92f156642cfbe0ccabafdadfdae, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/81a4ee9ec8974b0ea5db97dcb4c5dee5, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/ad858859ea894c4d9979643faec88cf7, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/defbeab378b0491e94b2b67da7f11639, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f6466eecbc7e487e88e5fea8f6cc3919, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/930c98242ebe4bb79eac67d96848a542, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/74ad9b58253648f9a15161781bdc9cde, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e71f2b41982943e29d4fba3191bf93fa, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a04afb56b7e14b82b17450d7e111791c, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0bb2c38300774699871b7c2229f0ba4f, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f0ba9413bf4e4ad697a4a504bad58748, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/bc2587f125064e40b010981326278a2e, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/feeaf874a7994a89acdcb28b32177ec7, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a9f3ff4eb64945889bdad6c248f9ab5a] to archive 2023-04-10 02:56:48,230 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-10 02:56:48,231 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/b9c48bd3b53c4fcba79b854a519a0942 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/b9c48bd3b53c4fcba79b854a519a0942 2023-04-10 02:56:48,232 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/9bf36dcb425846df93fc0df4ad232626 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/9bf36dcb425846df93fc0df4ad232626 2023-04-10 02:56:48,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0029e1d267ce420f8afe97b7a261cd5e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0029e1d267ce420f8afe97b7a261cd5e 2023-04-10 02:56:48,234 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/143aa7833e6e4a34aeffa274ff702a46 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/143aa7833e6e4a34aeffa274ff702a46 2023-04-10 02:56:48,234 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0a65f2b5736d4cedaa62c102b2397507 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0a65f2b5736d4cedaa62c102b2397507 2023-04-10 02:56:48,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a3c140390ce14fdca6ef3ce5e3098d71 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a3c140390ce14fdca6ef3ce5e3098d71 2023-04-10 02:56:48,236 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/21f5816f0f684da4a6c4d0ae4fa9bc1a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/21f5816f0f684da4a6c4d0ae4fa9bc1a 2023-04-10 02:56:48,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/492b161d71ca497d8be324fce2c0be9b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/492b161d71ca497d8be324fce2c0be9b 2023-04-10 02:56:48,238 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/eb3222a0a97d4f31acb213494b13475e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/eb3222a0a97d4f31acb213494b13475e 2023-04-10 02:56:48,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-10 02:56:48,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/fc48c85e63a7466da9bd4a66ac6999a9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/fc48c85e63a7466da9bd4a66ac6999a9 2023-04-10 02:56:48,240 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/51a979b2cd1d4429a9296729c0a25551 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/51a979b2cd1d4429a9296729c0a25551 2023-04-10 02:56:48,240 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/23c4f6301b874783a53c3716ede242e6 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/23c4f6301b874783a53c3716ede242e6 2023-04-10 02:56:48,241 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0b2096f66be64ebfa32d507ca3b76082 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0b2096f66be64ebfa32d507ca3b76082 2023-04-10 02:56:48,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e469a3b7bdde426f992e870a2d49da3d to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e469a3b7bdde426f992e870a2d49da3d 2023-04-10 02:56:48,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/7117bc4af6bf4806aa0b5a24f781ea5b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/7117bc4af6bf4806aa0b5a24f781ea5b 2023-04-10 02:56:48,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/58648913cf9d4c679de0bd9b0349a654 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/58648913cf9d4c679de0bd9b0349a654 2023-04-10 02:56:48,245 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e170e6a9a08b4c8693439a5db79648ef to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e170e6a9a08b4c8693439a5db79648ef 2023-04-10 02:56:48,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/40dfce2ed13d4daf81c4c1d524500f7a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/40dfce2ed13d4daf81c4c1d524500f7a 2023-04-10 02:56:48,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/fabc5eb5061141f3afdd44499e204036 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/fabc5eb5061141f3afdd44499e204036 2023-04-10 02:56:48,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/15246804a6db4cd3b7fdf5af293db746 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/15246804a6db4cd3b7fdf5af293db746 2023-04-10 02:56:48,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/af7ad25f5bfb414da2a71ce2c078d291 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/af7ad25f5bfb414da2a71ce2c078d291 2023-04-10 02:56:48,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e2a3b85ccaad44c3a3b1e08d3d0239b0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e2a3b85ccaad44c3a3b1e08d3d0239b0 2023-04-10 02:56:48,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/5c5282adc9464b928c391f041b37a87f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/5c5282adc9464b928c391f041b37a87f 2023-04-10 02:56:48,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/8892e5b8da9e43c3a9da36adbd942309 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/8892e5b8da9e43c3a9da36adbd942309 2023-04-10 02:56:48,254 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f3d3f92f156642cfbe0ccabafdadfdae to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f3d3f92f156642cfbe0ccabafdadfdae 2023-04-10 02:56:48,255 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/81a4ee9ec8974b0ea5db97dcb4c5dee5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/81a4ee9ec8974b0ea5db97dcb4c5dee5 2023-04-10 02:56:48,256 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/ad858859ea894c4d9979643faec88cf7 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/ad858859ea894c4d9979643faec88cf7 2023-04-10 02:56:48,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/defbeab378b0491e94b2b67da7f11639 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/defbeab378b0491e94b2b67da7f11639 2023-04-10 02:56:48,262 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f6466eecbc7e487e88e5fea8f6cc3919 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f6466eecbc7e487e88e5fea8f6cc3919 2023-04-10 02:56:48,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/930c98242ebe4bb79eac67d96848a542 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/930c98242ebe4bb79eac67d96848a542 2023-04-10 02:56:48,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/74ad9b58253648f9a15161781bdc9cde to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/74ad9b58253648f9a15161781bdc9cde 2023-04-10 02:56:48,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e71f2b41982943e29d4fba3191bf93fa to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/e71f2b41982943e29d4fba3191bf93fa 2023-04-10 02:56:48,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a04afb56b7e14b82b17450d7e111791c to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a04afb56b7e14b82b17450d7e111791c 2023-04-10 02:56:48,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0bb2c38300774699871b7c2229f0ba4f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/0bb2c38300774699871b7c2229f0ba4f 2023-04-10 02:56:48,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f0ba9413bf4e4ad697a4a504bad58748 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/f0ba9413bf4e4ad697a4a504bad58748 2023-04-10 02:56:48,267 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/bc2587f125064e40b010981326278a2e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/bc2587f125064e40b010981326278a2e 2023-04-10 02:56:48,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/feeaf874a7994a89acdcb28b32177ec7 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/feeaf874a7994a89acdcb28b32177ec7 2023-04-10 02:56:48,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a9f3ff4eb64945889bdad6c248f9ab5a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/a9f3ff4eb64945889bdad6c248f9ab5a 2023-04-10 02:56:48,274 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/recovered.edits/708.seqid, newMaxSeqId=708, maxSeqId=4 2023-04-10 02:56:48,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7. 2023-04-10 02:56:48,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6627f276a3252fdcc4f8b38ace724cf7: 2023-04-10 02:56:48,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:48,278 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=6627f276a3252fdcc4f8b38ace724cf7, regionState=CLOSED 2023-04-10 02:56:48,279 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681095408278"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681095408278"}]},"ts":"1681095408278"} 2023-04-10 02:56:48,281 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-10 02:56:48,281 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 6627f276a3252fdcc4f8b38ace724cf7, server=jenkins-hbase4.apache.org,38321,1681095222501 in 1.1380 sec 2023-04-10 02:56:48,282 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-10 02:56:48,282 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6627f276a3252fdcc4f8b38ace724cf7, UNASSIGN in 1.1430 sec 2023-04-10 02:56:48,282 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681095408282"}]},"ts":"1681095408282"} 2023-04-10 02:56:48,283 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-10 02:56:48,285 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-10 02:56:48,294 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1600 sec 2023-04-10 02:56:49,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-10 02:56:49,239 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-10 02:56:49,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-10 02:56:49,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:49,242 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:49,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-10 02:56:49,242 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:49,246 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:49,247 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C, FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/recovered.edits] 2023-04-10 02:56:49,250 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/09fddde5fe064f2397aebfc91020a9e2 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/09fddde5fe064f2397aebfc91020a9e2 2023-04-10 02:56:49,251 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/cc02ede091324e95afd26a6f3657799b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/A/cc02ede091324e95afd26a6f3657799b 2023-04-10 02:56:49,253 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/32798ea37e8347f7a400a2994c5ebf65 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/32798ea37e8347f7a400a2994c5ebf65 2023-04-10 02:56:49,253 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/9c466d9e95814c2fb4142c2afc03b798 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/B/9c466d9e95814c2fb4142c2afc03b798 2023-04-10 02:56:49,255 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/6f72e08ead46425ebfd289ae42178ddd to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/6f72e08ead46425ebfd289ae42178ddd 2023-04-10 02:56:49,256 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/ced234e27bdd4711a4d4ec02a83f6a39 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/C/ced234e27bdd4711a4d4ec02a83f6a39 2023-04-10 02:56:49,258 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/recovered.edits/708.seqid to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7/recovered.edits/708.seqid 2023-04-10 02:56:49,258 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/.tmp/data/default/TestAcidGuarantees/6627f276a3252fdcc4f8b38ace724cf7 2023-04-10 02:56:49,258 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-10 02:56:49,259 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-10 02:56:49,259 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-10 02:56:49,262 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304100d923db42c894497bd41eef050d51266 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304100d923db42c894497bd41eef050d51266 2023-04-10 02:56:49,262 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041011a767f9be1345a1beb8df91e2b132b5 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041011a767f9be1345a1beb8df91e2b132b5 2023-04-10 02:56:49,263 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041019245400175844649be0d299ea0b8c88 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041019245400175844649be0d299ea0b8c88 2023-04-10 02:56:49,264 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304104031e33c44404d38a95809294db56402 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304104031e33c44404d38a95809294db56402 2023-04-10 02:56:49,265 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041045c4998d0c594d02af5bb484e199962b to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041045c4998d0c594d02af5bb484e199962b 2023-04-10 02:56:49,266 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304104c2516da1ff24c8db197adfc23632834 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304104c2516da1ff24c8db197adfc23632834 2023-04-10 02:56:49,267 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041051a1bed015d742daabcfbcab44b5cc04 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041051a1bed015d742daabcfbcab44b5cc04 2023-04-10 02:56:49,267 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041054831c19ae4f4a7589e392df015f18a0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041054831c19ae4f4a7589e392df015f18a0 2023-04-10 02:56:49,268 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041054de7effe3ee4fd2b3df98c78c60c2cf to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041054de7effe3ee4fd2b3df98c78c60c2cf 2023-04-10 02:56:49,269 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304105685ce8deb6b40ada18ac456f88bba30 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304105685ce8deb6b40ada18ac456f88bba30 2023-04-10 02:56:49,270 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410620483d45a5d428b9ce14f4c70779fb7 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410620483d45a5d428b9ce14f4c70779fb7 2023-04-10 02:56:49,270 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041062c6e0661f2b4b0e83e5be2ddc1a6524 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041062c6e0661f2b4b0e83e5be2ddc1a6524 2023-04-10 02:56:49,271 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304106448505a9f234f31bd2c4e52de283ed8 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304106448505a9f234f31bd2c4e52de283ed8 2023-04-10 02:56:49,272 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410854c6c8a1b674cddacc10cc03c15de05 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410854c6c8a1b674cddacc10cc03c15de05 2023-04-10 02:56:49,272 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304108c5e353455b44c38bd35cdec0667678f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304108c5e353455b44c38bd35cdec0667678f 2023-04-10 02:56:49,273 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304108da37dcfa132409489ab4a5013d8ce0a to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304108da37dcfa132409489ab4a5013d8ce0a 2023-04-10 02:56:49,274 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410930bfaea024e4949b309c598d3142309 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410930bfaea024e4949b309c598d3142309 2023-04-10 02:56:49,275 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a050e0e1d97d492bb3d528406658b9f9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a050e0e1d97d492bb3d528406658b9f9 2023-04-10 02:56:49,275 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a1cb7d3d05164802a2de9a1e3f221457 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a1cb7d3d05164802a2de9a1e3f221457 2023-04-10 02:56:49,276 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a814c405504240c8b20781303e26ab54 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410a814c405504240c8b20781303e26ab54 2023-04-10 02:56:49,277 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410af04d073be844fccaf729d7bbdc64569 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410af04d073be844fccaf729d7bbdc64569 2023-04-10 02:56:49,278 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410bc095e10f1044004975ca9d1676ab9be to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410bc095e10f1044004975ca9d1676ab9be 2023-04-10 02:56:49,278 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410bc5ccf030d794d999460ae83ba60729e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410bc5ccf030d794d999460ae83ba60729e 2023-04-10 02:56:49,279 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410be34a5b86eca495398026765c18a4d48 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410be34a5b86eca495398026765c18a4d48 2023-04-10 02:56:49,280 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410bf71920d739f4c3d84d21299adb7e918 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410bf71920d739f4c3d84d21299adb7e918 2023-04-10 02:56:49,281 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410c83169eeac0246219b1c10f2f921dc78 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410c83169eeac0246219b1c10f2f921dc78 2023-04-10 02:56:49,282 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410cd24ae288f794883ae2190d16e5d6912 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410cd24ae288f794883ae2190d16e5d6912 2023-04-10 02:56:49,283 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410d605696ec40b4b40bc3f233f663851e9 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410d605696ec40b4b40bc3f233f663851e9 2023-04-10 02:56:49,284 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410d9ade8e5ca764b0a9d0c269526e0834f to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410d9ade8e5ca764b0a9d0c269526e0834f 2023-04-10 02:56:49,285 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410dda2610901034cceabb4f1ecd6d5e32e to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410dda2610901034cceabb4f1ecd6d5e32e 2023-04-10 02:56:49,286 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fc7c9f9262c547458506d34799650fa0 to hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230410fc7c9f9262c547458506d34799650fa0 2023-04-10 02:56:49,287 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-10 02:56:49,289 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:49,290 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-10 02:56:49,292 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-10 02:56:49,293 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:49,293 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-10 02:56:49,293 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681095409293"}]},"ts":"9223372036854775807"} 2023-04-10 02:56:49,294 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-10 02:56:49,294 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6627f276a3252fdcc4f8b38ace724cf7, NAME => 'TestAcidGuarantees,,1681095374161.6627f276a3252fdcc4f8b38ace724cf7.', STARTKEY => '', ENDKEY => ''}] 2023-04-10 02:56:49,295 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-10 02:56:49,295 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681095409295"}]},"ts":"9223372036854775807"} 2023-04-10 02:56:49,296 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-10 02:56:49,297 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-10 02:56:49,298 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 57 msec 2023-04-10 02:56:49,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36119] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-10 02:56:49,343 INFO [Listener at localhost/44533] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-10 02:56:49,356 INFO [Listener at localhost/44533] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 227), OpenFileDescriptor=521 (was 561), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=419 (was 463), ProcessCount=169 (was 180), AvailableMemoryMB=1659 (was 2375) 2023-04-10 02:56:49,357 INFO [Listener at localhost/44533] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-10 02:56:49,357 INFO [Listener at localhost/44533] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-10 02:56:49,357 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ddd91d3 to 127.0.0.1:60568 2023-04-10 02:56:49,358 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:49,358 DEBUG [Listener at localhost/44533] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-10 02:56:49,358 DEBUG [Listener at localhost/44533] util.JVMClusterUtil(257): Found active master hash=696616797, stopped=false 2023-04-10 02:56:49,358 INFO [Listener at localhost/44533] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,36119,1681095221303 2023-04-10 02:56:49,359 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-10 02:56:49,359 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-10 02:56:49,359 INFO [Listener at localhost/44533] procedure2.ProcedureExecutor(629): Stopping 2023-04-10 02:56:49,359 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-10 02:56:49,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-10 02:56:49,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-10 02:56:49,360 DEBUG [Listener at localhost/44533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f4adb4d to 127.0.0.1:60568 2023-04-10 02:56:49,360 DEBUG [Listener at localhost/44533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:49,361 INFO [Listener at localhost/44533] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38321,1681095222501' ***** 2023-04-10 02:56:49,361 INFO [Listener at localhost/44533] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-10 02:56:49,361 INFO [RS:0;jenkins-hbase4:38321] regionserver.HeapMemoryManager(220): Stopping 2023-04-10 02:56:49,361 INFO [RS:0;jenkins-hbase4:38321] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-10 02:56:49,361 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-10 02:56:49,361 INFO [RS:0;jenkins-hbase4:38321] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-10 02:56:49,362 INFO [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(3304): Received CLOSE for 78f9d525c18aba75e4f5b4c3629f0ae0 2023-04-10 02:56:49,362 INFO [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:49,362 DEBUG [RS:0;jenkins-hbase4:38321] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x341e8eb4 to 127.0.0.1:60568 2023-04-10 02:56:49,362 DEBUG [RS:0;jenkins-hbase4:38321] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:49,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 78f9d525c18aba75e4f5b4c3629f0ae0, disabling compactions & flushes 2023-04-10 02:56:49,363 INFO [RS:0;jenkins-hbase4:38321] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-10 02:56:49,363 INFO [RS:0;jenkins-hbase4:38321] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-10 02:56:49,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. 2023-04-10 02:56:49,363 INFO [RS:0;jenkins-hbase4:38321] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-10 02:56:49,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. 2023-04-10 02:56:49,363 INFO [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-10 02:56:49,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. after waiting 0 ms 2023-04-10 02:56:49,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. 2023-04-10 02:56:49,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 78f9d525c18aba75e4f5b4c3629f0ae0 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-10 02:56:49,363 INFO [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-10 02:56:49,363 DEBUG [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1479): Online Regions={78f9d525c18aba75e4f5b4c3629f0ae0=hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0., 1588230740=hbase:meta,,1.1588230740} 2023-04-10 02:56:49,366 DEBUG [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1505): Waiting on 1588230740, 78f9d525c18aba75e4f5b4c3629f0ae0 2023-04-10 02:56:49,366 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-10 02:56:49,366 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-10 02:56:49,366 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-10 02:56:49,366 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-10 02:56:49,366 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-10 02:56:49,366 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-10 02:56:49,369 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-10 02:56:49,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/namespace/78f9d525c18aba75e4f5b4c3629f0ae0/.tmp/info/923beaafde62475c8c0255f707bc863f 2023-04-10 02:56:49,377 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/.tmp/info/b5f5d0eb7f684dcbaae4e44c6ded650a 2023-04-10 02:56:49,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/namespace/78f9d525c18aba75e4f5b4c3629f0ae0/.tmp/info/923beaafde62475c8c0255f707bc863f as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/namespace/78f9d525c18aba75e4f5b4c3629f0ae0/info/923beaafde62475c8c0255f707bc863f 2023-04-10 02:56:49,381 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b5f5d0eb7f684dcbaae4e44c6ded650a 2023-04-10 02:56:49,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/namespace/78f9d525c18aba75e4f5b4c3629f0ae0/info/923beaafde62475c8c0255f707bc863f, entries=2, sequenceid=6, filesize=4.8 K 2023-04-10 02:56:49,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 78f9d525c18aba75e4f5b4c3629f0ae0 in 20ms, sequenceid=6, compaction requested=false 2023-04-10 02:56:49,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/namespace/78f9d525c18aba75e4f5b4c3629f0ae0/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-10 02:56:49,389 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. 2023-04-10 02:56:49,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 78f9d525c18aba75e4f5b4c3629f0ae0: 2023-04-10 02:56:49,389 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/.tmp/rep_barrier/0edefb2a05e540dd9b53518f96f217a8 2023-04-10 02:56:49,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681095224771.78f9d525c18aba75e4f5b4c3629f0ae0. 2023-04-10 02:56:49,391 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0edefb2a05e540dd9b53518f96f217a8 2023-04-10 02:56:49,566 DEBUG [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-10 02:56:49,766 DEBUG [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-10 02:56:49,801 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/.tmp/table/56340bdc4c6f4453a589b64b0cc908e1 2023-04-10 02:56:49,804 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 56340bdc4c6f4453a589b64b0cc908e1 2023-04-10 02:56:49,805 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/.tmp/info/b5f5d0eb7f684dcbaae4e44c6ded650a as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/info/b5f5d0eb7f684dcbaae4e44c6ded650a 2023-04-10 02:56:49,807 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b5f5d0eb7f684dcbaae4e44c6ded650a 2023-04-10 02:56:49,807 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/info/b5f5d0eb7f684dcbaae4e44c6ded650a, entries=22, sequenceid=93, filesize=7.1 K 2023-04-10 02:56:49,809 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/.tmp/rep_barrier/0edefb2a05e540dd9b53518f96f217a8 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/rep_barrier/0edefb2a05e540dd9b53518f96f217a8 2023-04-10 02:56:49,811 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0edefb2a05e540dd9b53518f96f217a8 2023-04-10 02:56:49,811 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/rep_barrier/0edefb2a05e540dd9b53518f96f217a8, entries=6, sequenceid=93, filesize=5.5 K 2023-04-10 02:56:49,812 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/.tmp/table/56340bdc4c6f4453a589b64b0cc908e1 as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/table/56340bdc4c6f4453a589b64b0cc908e1 2023-04-10 02:56:49,814 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 56340bdc4c6f4453a589b64b0cc908e1 2023-04-10 02:56:49,815 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/table/56340bdc4c6f4453a589b64b0cc908e1, entries=9, sequenceid=93, filesize=5.5 K 2023-04-10 02:56:49,815 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 449ms, sequenceid=93, compaction requested=false 2023-04-10 02:56:49,822 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-10 02:56:49,822 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-10 02:56:49,823 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-10 02:56:49,823 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-10 02:56:49,823 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-10 02:56:49,967 INFO [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38321,1681095222501; all regions closed. 2023-04-10 02:56:49,974 DEBUG [RS:0;jenkins-hbase4:38321] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/oldWALs 2023-04-10 02:56:49,974 INFO [RS:0;jenkins-hbase4:38321] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38321%2C1681095222501.meta:.meta(num 1681095224528) 2023-04-10 02:56:49,978 DEBUG [RS:0;jenkins-hbase4:38321] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/oldWALs 2023-04-10 02:56:49,978 INFO [RS:0;jenkins-hbase4:38321] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38321%2C1681095222501:(num 1681095224366) 2023-04-10 02:56:49,978 DEBUG [RS:0;jenkins-hbase4:38321] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:49,978 INFO [RS:0;jenkins-hbase4:38321] regionserver.LeaseManager(133): Closed leases 2023-04-10 02:56:49,979 INFO [RS:0;jenkins-hbase4:38321] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-10 02:56:49,979 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-10 02:56:49,980 INFO [RS:0;jenkins-hbase4:38321] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38321 2023-04-10 02:56:49,985 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38321,1681095222501 2023-04-10 02:56:49,985 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-10 02:56:49,985 ERROR [Listener at localhost/44533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2d56301 rejected from java.util.concurrent.ThreadPoolExecutor@6eae783[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2432] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-10 02:56:49,986 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-10 02:56:49,986 ERROR [Listener at localhost/44533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5b8500a0 rejected from java.util.concurrent.ThreadPoolExecutor@6eae783[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2432] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-10 02:56:49,986 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,38321,1681095222501] 2023-04-10 02:56:49,987 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,38321,1681095222501; numProcessing=1 2023-04-10 02:56:49,989 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,38321,1681095222501 already deleted, retry=false 2023-04-10 02:56:49,989 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,38321,1681095222501 expired; onlineServers=0 2023-04-10 02:56:49,989 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36119,1681095221303' ***** 2023-04-10 02:56:49,989 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-10 02:56:49,989 DEBUG [M:0;jenkins-hbase4:36119] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68b88d66, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-10 02:56:49,989 INFO [M:0;jenkins-hbase4:36119] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36119,1681095221303 2023-04-10 02:56:49,989 INFO [M:0;jenkins-hbase4:36119] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36119,1681095221303; all regions closed. 2023-04-10 02:56:49,989 DEBUG [M:0;jenkins-hbase4:36119] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-10 02:56:49,989 DEBUG [M:0;jenkins-hbase4:36119] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-10 02:56:49,990 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-10 02:56:49,990 DEBUG [M:0;jenkins-hbase4:36119] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-10 02:56:49,990 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681095224031] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681095224031,5,FailOnTimeoutGroup] 2023-04-10 02:56:49,990 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681095224031] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681095224031,5,FailOnTimeoutGroup] 2023-04-10 02:56:49,990 INFO [M:0;jenkins-hbase4:36119] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-10 02:56:49,990 INFO [M:0;jenkins-hbase4:36119] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-10 02:56:49,990 INFO [M:0;jenkins-hbase4:36119] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-10 02:56:49,990 DEBUG [M:0;jenkins-hbase4:36119] master.HMaster(1512): Stopping service threads 2023-04-10 02:56:49,990 INFO [M:0;jenkins-hbase4:36119] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-10 02:56:49,990 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-10 02:56:49,990 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-10 02:56:49,991 ERROR [M:0;jenkins-hbase4:36119] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-10 02:56:49,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-10 02:56:49,991 INFO [M:0;jenkins-hbase4:36119] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-10 02:56:49,991 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-10 02:56:49,991 DEBUG [M:0;jenkins-hbase4:36119] zookeeper.ZKUtil(398): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-10 02:56:49,992 WARN [M:0;jenkins-hbase4:36119] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-10 02:56:49,992 INFO [M:0;jenkins-hbase4:36119] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-10 02:56:49,992 INFO [M:0;jenkins-hbase4:36119] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-10 02:56:49,992 DEBUG [M:0;jenkins-hbase4:36119] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-10 02:56:49,992 INFO [M:0;jenkins-hbase4:36119] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-10 02:56:49,992 DEBUG [M:0;jenkins-hbase4:36119] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-10 02:56:49,992 DEBUG [M:0;jenkins-hbase4:36119] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-10 02:56:49,992 DEBUG [M:0;jenkins-hbase4:36119] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-10 02:56:49,992 INFO [M:0;jenkins-hbase4:36119] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.08 KB heapSize=393.75 KB 2023-04-10 02:56:50,088 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-10 02:56:50,088 INFO [RS:0;jenkins-hbase4:38321] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38321,1681095222501; zookeeper connection closed. 2023-04-10 02:56:50,088 ERROR [Listener at localhost/44533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@23669a9a rejected from java.util.concurrent.ThreadPoolExecutor@6eae783[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2432] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-10 02:56:50,088 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): regionserver:38321-0x1016bfd46a00001, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-10 02:56:50,089 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1eb28efa] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1eb28efa 2023-04-10 02:56:50,089 ERROR [Listener at localhost/44533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@503402ad rejected from java.util.concurrent.ThreadPoolExecutor@6eae783[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2432] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-10 02:56:50,089 INFO [Listener at localhost/44533] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-10 02:56:50,405 INFO [M:0;jenkins-hbase4:36119] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.08 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f738ffe7dd9d4d04b8d8b04c5503340e 2023-04-10 02:56:50,408 INFO [M:0;jenkins-hbase4:36119] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f738ffe7dd9d4d04b8d8b04c5503340e 2023-04-10 02:56:50,409 DEBUG [M:0;jenkins-hbase4:36119] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f738ffe7dd9d4d04b8d8b04c5503340e as hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f738ffe7dd9d4d04b8d8b04c5503340e 2023-04-10 02:56:50,411 INFO [M:0;jenkins-hbase4:36119] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f738ffe7dd9d4d04b8d8b04c5503340e 2023-04-10 02:56:50,411 INFO [M:0;jenkins-hbase4:36119] regionserver.HStore(1080): Added hdfs://localhost:42593/user/jenkins/test-data/e57a2ccd-aa91-600a-9655-c5cac0f76717/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f738ffe7dd9d4d04b8d8b04c5503340e, entries=65, sequenceid=578, filesize=24.7 K 2023-04-10 02:56:50,412 INFO [M:0;jenkins-hbase4:36119] regionserver.HRegion(2948): Finished flush of dataSize ~345.08 KB/353359, heapSize ~393.73 KB/403184, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 420ms, sequenceid=578, compaction requested=false 2023-04-10 02:56:50,413 INFO [M:0;jenkins-hbase4:36119] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-10 02:56:50,413 DEBUG [M:0;jenkins-hbase4:36119] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-10 02:56:50,416 INFO [M:0;jenkins-hbase4:36119] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-10 02:56:50,416 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-10 02:56:50,416 INFO [M:0;jenkins-hbase4:36119] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36119 2023-04-10 02:56:50,418 DEBUG [M:0;jenkins-hbase4:36119] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,36119,1681095221303 already deleted, retry=false 2023-04-10 02:56:50,519 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-10 02:56:50,519 INFO [M:0;jenkins-hbase4:36119] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36119,1681095221303; zookeeper connection closed. 2023-04-10 02:56:50,519 ERROR [Listener at localhost/44533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4af1cf36 rejected from java.util.concurrent.ThreadPoolExecutor@604fe08d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4068] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-10 02:56:50,519 DEBUG [Listener at localhost/44533-EventThread] zookeeper.ZKWatcher(600): master:36119-0x1016bfd46a00000, quorum=127.0.0.1:60568, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-10 02:56:50,519 ERROR [Listener at localhost/44533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@20d3e23e rejected from java.util.concurrent.ThreadPoolExecutor@604fe08d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4068] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-10 02:56:50,521 WARN [Listener at localhost/44533] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-10 02:56:50,524 INFO [Listener at localhost/44533] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-10 02:56:50,635 WARN [BP-909374272-172.31.14.131-1681095217954 heartbeating to localhost/127.0.0.1:42593] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-10 02:56:50,635 WARN [BP-909374272-172.31.14.131-1681095217954 heartbeating to localhost/127.0.0.1:42593] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-909374272-172.31.14.131-1681095217954 (Datanode Uuid 3badec8f-43a8-47c0-ae8c-646406d4f0bc) service to localhost/127.0.0.1:42593 2023-04-10 02:56:50,639 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/cluster_55f563f5-c06c-630a-24a5-87501b65e74d/dfs/data/data1/current/BP-909374272-172.31.14.131-1681095217954] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-10 02:56:50,640 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bf471e1f-805e-d234-5b19-b3cab63a9089/cluster_55f563f5-c06c-630a-24a5-87501b65e74d/dfs/data/data2/current/BP-909374272-172.31.14.131-1681095217954] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-10 02:56:50,686 INFO [Listener at localhost/44533] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-10 02:56:50,862 INFO [Listener at localhost/44533] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-10 02:56:51,040 INFO [Listener at localhost/44533] hbase.HBaseTestingUtility(1293): Minicluster is down